CatchUp

Catching Up on CatchUp: A Dagger-powered Plugin System

Zac Sweers

Nov 13, 2017 • 10 min read

For context — see the introduction post. This is part of a series of technical deep dives into different things I’ve learned along the way.

This post assumes some prior knowledge of Dagger.

In CatchUp, the principle architecture is that you have a set of services available for consumption (Hacker News, Reddit, Dribbble, etc). These services could:

Be text or visual
Sometimes support paging (numeric or cursor)
Have different themes
Powered by different APIs (Firebase, REST, GraphQL, etc)
Ideally also has client-side caching

CatchUp’s architecture has gone through a number of iterations to support this, each building on the previous and factoring in what did and didn’t work well. The final result is something I’m pretty happy with, a Dagger-powered plugin system where services are totally isolated and Dagger does the heavy lifting in wiring them up automatically. I’ll dive more into that farther down, but I think it’s important to detail how it got there.

v1: Groundwork

In proper side project fashion, most of CatchUp’s services were written totally procedurally. Each was a custom Conductor Controller implementation that was just added as a child controller to the main ViewPager. Started with HackerNews and Reddit, and more services were added periodically after that. After these first two, it was clear that refactoring for reuse was needed to help it scale and not duplicate code. This led to the sort of v1 architecture: BaseNewsController.

BaseNewsController was pretty simple. It handled the the basic list boilerplate for showing items (setting up the RecyclerView, adapter, fetching, loading/display/error states, etc) and had some abstract methods that subclasses should implement to return streams of data. Pretty straightforward and most of the the services were implemented as subclasses of this. It had convenience context theming methods and exposed a generic fun getDataSingle(): Single<List<T>> for fetching data.

At some point along the way, Dribbble support was added. Dribbble’s UI is totally visual, and while it shares some mechanics, its controller ended up being a totally custom implementation to properly support it. This was an early indicator that things could be done better. When new APIs like paging were added later, it needed to be duplicated across both Dribbble as well as BaseNewsController. Some stuff was extracted to a base class called ServiceController, but it was a stopgap at best.

What worked well:

Beginning to stencil expectations of services. Fetching mechanism, service metadata, theming, etc.
Core UIs were mostly solid and standardized, from controller layout to individual items.

What didn’t work well:

No saved instance state or local storage story
Each service is its own snowflake. Maintenance is harder, each responsible for own presentation logic (often duplicated)
Relies on awkward inheritance
Text and visual not reconciled
Themes were handled in XML styles. While convenient for static themes, this wouldn’t scale with dynamically added services. This also didn’t allow for shared ViewPools since it relied on context theming.

v1.5: (Bolting on) Local Storage

Eventually I wanted to add local storage support. This probably merits its own blogpost, but the gist of the story is that I tried a bunch of different libraries and patterns and some core components materialized along the way:

Having unique models from every endpoint wasn’t going to scale. What I needed was a single model that reflected my UI. This led to CatchUpItem, and the pattern of services mapping their items to this.
With a consistent UI model, suddenly the storage story became much simpler. I didn’t have to write storage for individual services, just for CatchUpItem.
I ultimately settled on Room from the architecture components because of its simplicity, support for custom type converters, RxJava support, Kotlin data class support, and good threading story.
This storage layer simply decorated a real service and orchestrated fetches from disk or network.

This led to ~v1.5 of the architecture with the introduction of StorageBackedController as an orchestrator for this. It extended BaseNewsController (I know, I know), used the same API semantics but exposed its own abstract fetch method fun fetchData(page: Int): Single<List<CatchUpItem>>. Subclasses then implemented this to return their data and convert it to CatchUpItems. A neat thing was that this was the first bit of instance state handling in CatchUp. This controller could manage instance state, track pages, and intelligently backfill pages (with some slick Rx swizzling) on configuration changes.

Medium, Product Hunt, GitHub, and DesignerNews controllers were all moved to this pattern. I held off on other Controllers because it became clear to me at some point that I wanted to do something better.

What worked well:

Local storage, save instance state story, backfilling, neat stuff
Unified view model type to put in storage simplified API

What didn’t work well:

Still relies on awkward inheritance
Only supported numeric pages
Instance state handling was buggy
Text and visual still not reconciled
Theming still not scalable
Things were really fragmented

v2: Service Interface

Over the course of the various iterations above, a picture began to form of where I wanted to end up. Some core principles that evolved out of it:

Each service should return CatchUpItems. What they did under the hood was irrelevant, as long as they mapped it to CatchUpItem, I could handle the UI presentation and storage layer separately.
Services should be decoupled from UI; Not be controllers. They should instead be just plain old objects that implement a Service interface. This should also end the inheritance party, and the allowance of composition pushes storage logic into a simple delegating implementation that wraps a real one.
Resolve text and visual services behind a unified interface. The idea is that any service should just indicate whether it’s text or visual with mostly the same information, and the rest is just left up to the UI for how to present it. Adding visual metadata to CatchUpItem should be enough for this, and then binding logic for text or visual list items is easy enough. The controller can handle which adapter to show in its RecyclerView and all the rest can be reused.

The vision for this was pretty simple: create a Service interface (and its granular components), make services implement it, and make the app consume them in an abstracted manner. An added goal was I wanted to try decoupling at the Gradle module level long term, for isolation as well as build perf reasons. The API was written into its own package at first before being pulled out into an api module that :app depends on (if you’re refactoring libraries out of your code, I highly recommend using this from-the-bottom approach). By having API in a separate module, the app could depend on it and provide integration points for it, while any implementers of the service could also depend on it to provide implementations. This also means that any number of integration points could support the API too, which opens the doors for alternative implementations (different UIs, form factors, etc).

So you’ve got n number of Services with their String IDs (Reddit -> "reddit", Hacker News -> "hn", etc) . These are represented as a Map<String, Provider<Service>> at the consumer-level (namely in the PagerController class that displays all of them).

Note the Provider here, which is our first sight of Dagger making an appearance. The idea here is that these are lazily loaded by the pager based on which pages are supposed to be visible. For presenting the UI, there’s a new ServiceController class (spiritual successor to all the previous ones) that’s used to display a given service for a page. The services map is @Injected into it downstream in the DI graph, and the controller itself just gets a String argument for which ID key to use in the graph.

Quick note before we continue: In Kotlin, you have to add @JvmSuppressWildcards to most injection points that have generics for Dagger to correctly link. I’ve omitted them here for readability.

The Dagger-fu

Here’s the fun part. That Map? It’s @Injected as a Dagger Multibinding. Those services? Constructor-injectable now that they’re regular non-UI classes.

Service injection in ServiceController.kt

We can define every service in the context of their own Dagger module and provide that service with a @MapKey annotation key’d by their ID into a multibound Services map.

That map is what becomes the Map<String, Provider<Service>> injection site we saw earlier. Even better, you can provide regular services, declare that Provider designation at your injection point, and Dagger will automatically wrap them in providers for full laziness. What’s this look like in code?

First we define our integration point in our app. This sets up our multibinding.

@Module
abstract class IntegrationModule {
 
  @Multibinds
  abstract fun services(): Map<String, Service>
}

Next we’ll stub some example FooService and its Dagger module.

internal class FooService @Inject constructor(...) : Service {
  // Implementation
}
@Module
internal class FooModule {
  @IntoMap  
  @ServiceMetaKey("fooServiceId")  
  @Binds  
  internal abstract fun fooService(service: FooService): Service
}

Finally, to connect the dots, we’ll need to add the FooModule to the list of included modules to our IntegrationModule.

@Module(includes = [FooModule::class])
abstract class IntegrationModule {
 
  @Multibinds
  abstract fun services(): Map<String, Service>
}

Boom, we’re in business. This is a slimmed down version of what CatchUp does, but hopefully you get the gist. With this setup, we’re able to easily contribute services into the IntegrationModule, while letting Dagger do all the plumbing and instance handling.

Revisiting local storage

So we’ve got a functioning plugin system with multibindings. What happened to local storage though? Now that services are abstracted out behind this common façade and easily injectable, we can leverage Dagger further here. Borrowing from the early work of just intelligent delegation, StorageBackedService was written as a Service implementation that just wraps a real Service and orchestrates between fetching from the composed service or the Room DB. It also fixes a bunch of the pitfalls from before with support for token-based page backfilling (more slick RxJava swizzling) and an improved DataRequest API.

The trick here is getting this into Dagger. We don’t want to lose the Provider handling, but we also need to get access to the Services before they’re returned so we can wrap them up in our storage-backed one. I chose to handle this with a Dagger qualifier and inner provision. This put one layer of indirection between services provided and their eventual injection, but this is ok. Here’s how this looks:

@Qualifier internal annotation class FinalServices
@Module(includes = [FooModule::class])
abstract class IntegrationModule {
 
  @Multibinds   
  abstract fun services(): Map<String, Service> . 
companion object {
    @Provides
    @JvmStatic
    @FinalServices
    fun provideFinalServices(
        services: Map<String, Provider<Service>>)
        : Map<String, Provider<Service>> {
      return services.mapValues {
        Provider<Service> { StorageBackedService(it.get()) }
      }
    }
  }
}

What’s going on here? We’re taking in the mapped (provider’d) Services here and returning a new map of the same signature whose values, when called, fetch the real service from the input map and wraps them in a StorageBackedService. To get this updated map, we simply add the qualifier to the injection point in ServiceController when we consume it. This is exactly how it’s done in CatchUp.

Now we’re injecting our storage-aware services, all still with the lazy Provider approaches and just a little bit of swizzling under the hood to get there.

Forced isolation

Farther up, I mentioned wanting to eventually move services to their own Gradle subproject. Dagger will generate factories for each dagger module, regardless of whether or not they’re used within a given compilation. What this means is that you can define your modules in any subproject, run Dagger’s processor over them during compilation, and consume them in another project (in the conventional case — an individual Gradle project, jar, etc). Dagger will also understand these modules if you include them into a subproject in your local project by generating code that expects their factories. What this means is that each service can live entirely in its own subproject, fully self-contained and only exposing its to-be-included Dagger module.

So when we have this code in CatchUp:

@Module(includes = [FooModule::class])
abstract class IntegrationModule {
  // ...
}

FooModule can actually be compiled and imported from a completely different project! In CatchUp, every service implementation lives in its own gradle subproject. This helps with isolation, parallelism, and in general keeping the service-api module clean and focused to minimize induced bloat on services. This leaves a nice project structure like this:

:app
  - src/main/kotlin/.../IntegrationModule.kt
:service-api
:services
  :github
  :dribbble
  :reddit
  //... etc

Final icing on the cake — in each service, the only public element is the Dagger module class itself. Everything else (including its inner methods, the service implementation, etc) is Kotlin’s internal visibility, which really drives home the isolation and low API surface area. The integration point literally can’t know anything about a service’s internal APIs — the compiler won’t let it!

Conclusion

I’m pretty happy with how things ended up. Once the Service architecture became concrete, things really fell into place quickly. I was able to migrate all the services over the course of about two days with relative ease, tweaking some things here and there as they came up. Some other benefits along the way were that it becomes trivial to add new services to it, and it’s easy to test (if I wrote them… #sideprojecting).

Future

Is this good now? I think so. Is it done? Definitely not. There’s a few things next on the horizon with this architecture

Manually having to wire up each downstream module to the integration point is a bit annoying and tedious. I have some ideas of how to automate this with code gen that I hope to get done in the next couple weeks.
The Service API can be opened to be more flexible and add support for new features. One such feature I plan to add next is the ability for services to specify their own preferences file in xml resources via field in ServiceMeta. CatchUp itself could just read this off and show the inflated xml in a generic preferences container for that key in settings, abstracted away from the content details.
Having the UI so decoupled from implementation details means that prototyping different UI patterns now becomes very easy. Two things I want to try next are some sort of tablet UI story and Litho.
I think this same system could be leveraged to allow services to provide their own debugging tools, such as mock responses/data.