Mastodon

Catching Up on CatchUp: A Dagger-powered Plugin System

For context — see the introduction post. This is part of a series of technical deep dives into different things I’ve learned along the way.

This post assumes some prior knowledge of Dagger.

In CatchUp, the principle architecture is that you have a set of services available for consumption (Hacker News, Reddit, Dribbble, etc). These services could:

CatchUp’s architecture has gone through a number of iterations to support this, each building on the previous and factoring in what did and didn’t work well. The final result is something I’m pretty happy with, a Dagger-powered plugin system where services are totally isolated and Dagger does the heavy lifting in wiring them up automatically. I’ll dive more into that farther down, but I think it’s important to detail how it got there.

v1: Groundwork

In proper side project fashion, most of CatchUp’s services were written totally procedurally. Each was a custom Conductor Controller implementation that was just added as a child controller to the main ViewPager. Started with HackerNews and Reddit, and more services were added periodically after that. After these first two, it was clear that refactoring for reuse was needed to help it scale and not duplicate code. This led to the sort of v1 architecture: BaseNewsController.

BaseNewsController was pretty simple. It handled the the basic list boilerplate for showing items (setting up the RecyclerView, adapter, fetching, loading/display/error states, etc) and had some abstract methods that subclasses should implement to return streams of data. Pretty straightforward and most of the the services were implemented as subclasses of this. It had convenience context theming methods and exposed a generic fun getDataSingle(): Single<List<T>> for fetching data.

At some point along the way, Dribbble support was added. Dribbble’s UI is totally visual, and while it shares some mechanics, its controller ended up being a totally custom implementation to properly support it. This was an early indicator that things could be done better. When new APIs like paging were added later, it needed to be duplicated across both Dribbble as well as BaseNewsController. Some stuff was extracted to a base class called ServiceController, but it was a stopgap at best.

What worked well:

What didn’t work well:

v1.5: (Bolting on) Local Storage

Eventually I wanted to add local storage support. This probably merits its own blogpost, but the gist of the story is that I tried a bunch of different libraries and patterns and some core components materialized along the way:

This led to ~v1.5 of the architecture with the introduction of StorageBackedController as an orchestrator for this. It extended BaseNewsController (I know, I know), used the same API semantics but exposed its own abstract fetch method fun fetchData(page: Int): Single<List<CatchUpItem>>. Subclasses then implemented this to return their data and convert it to CatchUpItems. A neat thing was that this was the first bit of instance state handling in CatchUp. This controller could manage instance state, track pages, and intelligently backfill pages (with some slick Rx swizzling) on configuration changes.

Medium, Product Hunt, GitHub, and DesignerNews controllers were all moved to this pattern. I held off on other Controllers because it became clear to me at some point that I wanted to do something better.

What worked well:

What didn’t work well:

Spaghetti architecture

v2: Service Interface

Over the course of the various iterations above, a picture began to form of where I wanted to end up. Some core principles that evolved out of it:

The vision for this was pretty simple: create a Service interface (and its granular components), make services implement it, and make the app consume them in an abstracted manner. An added goal was I wanted to try decoupling at the Gradle module level long term, for isolation as well as build perf reasons. The API was written into its own package at first before being pulled out into an api module that :app depends on (if you’re refactoring libraries out of your code, I highly recommend using this from-the-bottom approach). By having API in a separate module, the app could depend on it and provide integration points for it, while any implementers of the service could also depend on it to provide implementations. This also means that any number of integration points could support the API too, which opens the doors for alternative implementations (different UIs, form factors, etc).

The Service interface in Service.kt

So you’ve got n number of Services with their String IDs (Reddit -> "reddit", Hacker News -> "hn", etc) . These are represented as a Map<String, Provider<Service>> at the consumer-level (namely in the PagerController class that displays all of them).

Note the Provider here, which is our first sight of Dagger making an appearance. The idea here is that these are lazily loaded by the pager based on which pages are supposed to be visible. For presenting the UI, there’s a new ServiceController class (spiritual successor to all the previous ones) that’s used to display a given service for a page. The services map is @Injected into it downstream in the DI graph, and the controller itself just gets a String argument for which ID key to use in the graph.


Quick note before we continue: In Kotlin, you have to add @JvmSuppressWildcards to most injection points that have generics for Dagger to correctly link. I’ve omitted them here for readability.


The Dagger-fu

Here’s the fun part. That Map? It’s @Injected as a Dagger Multibinding. Those services? Constructor-injectable now that they’re regular non-UI classes.

Service injection in ServiceController.kt

We can define every service in the context of their own Dagger module and provide that service with a @MapKey annotation key’d by their ID into a multibound Services map.

That map is what becomes the Map<String, Provider<Service>> injection site we saw earlier. Even better, you can provide regular services, declare that Provider designation at your injection point, and Dagger will automatically wrap them in providers for full laziness. What’s this look like in code?

First we define our integration point in our app. This sets up our multibinding.

@Module
abstract class IntegrationModule {
 
  @Multibinds
  abstract fun services(): Map<String, Service>
}

Next we’ll stub some example FooService and its Dagger module.

internal class FooService @Inject constructor(...) : Service {
  // Implementation
}
@Module
internal class FooModule {
  @IntoMap  
  @ServiceMetaKey("fooServiceId")  
  @Binds  
  internal abstract fun fooService(service: FooService): Service
}

Finally, to connect the dots, we’ll need to add the FooModule to the list of included modules to our IntegrationModule.

@Module(includes = [FooModule::class])
abstract class IntegrationModule {
 
  @Multibinds
  abstract fun services(): Map<String, Service>
}

Boom, we’re in business. This is a slimmed down version of what CatchUp does, but hopefully you get the gist. With this setup, we’re able to easily contribute services into the IntegrationModule, while letting Dagger do all the plumbing and instance handling.

Revisiting local storage

So we’ve got a functioning plugin system with multibindings. What happened to local storage though? Now that services are abstracted out behind this common façade and easily injectable, we can leverage Dagger further here. Borrowing from the early work of just intelligent delegation, StorageBackedService was written as a Service implementation that just wraps a real Service and orchestrates between fetching from the composed service or the Room DB. It also fixes a bunch of the pitfalls from before with support for token-based page backfilling (more slick RxJava swizzling) and an improved DataRequest API.

The trick here is getting this into Dagger. We don’t want to lose the Provider handling, but we also need to get access to the Services before they’re returned so we can wrap them up in our storage-backed one. I chose to handle this with a Dagger qualifier and inner provision. This put one layer of indirection between services provided and their eventual injection, but this is ok. Here’s how this looks:

@Qualifier internal annotation class FinalServices
@Module(includes = [FooModule::class])
abstract class IntegrationModule {
 
  @Multibinds   
  abstract fun services(): Map<String, Service> . 
companion object {
    @Provides
    @JvmStatic
    @FinalServices
    fun provideFinalServices(
        services: Map<String, Provider<Service>>)
        : Map<String, Provider<Service>> {
      return services.mapValues {
        Provider<Service> { StorageBackedService(it.get()) }
      }
    }
  }
}

What’s going on here? We’re taking in the mapped (provider’d) Services here and returning a new map of the same signature whose values, when called, fetch the real service from the input map and wraps them in a StorageBackedService. To get this updated map, we simply add the qualifier to the injection point in ServiceController when we consume it. This is exactly how it’s done in CatchUp.

Now we’re injecting our storage-aware services, all still with the lazy Provider approaches and just a little bit of swizzling under the hood to get there.

Service injection in ServiceController.kt

Forced isolation

Farther up, I mentioned wanting to eventually move services to their own Gradle subproject. Dagger will generate factories for each dagger module, regardless of whether or not they’re used within a given compilation. What this means is that you can define your modules in any subproject, run Dagger’s processor over them during compilation, and consume them in another project (in the conventional case — an individual Gradle project, jar, etc). Dagger will also understand these modules if you include them into a subproject in your local project by generating code that expects their factories. What this means is that each service can live entirely in its own subproject, fully self-contained and only exposing its to-be-included Dagger module.

So when we have this code in CatchUp:

@Module(includes = [FooModule::class])
abstract class IntegrationModule {
  // ...
}

FooModule can actually be compiled and imported from a completely different project! In CatchUp, every service implementation lives in its own gradle subproject. This helps with isolation, parallelism, and in general keeping the service-api module clean and focused to minimize induced bloat on services. This leaves a nice project structure like this:

:app
  - src/main/kotlin/.../IntegrationModule.kt
:service-api
:services
  :github
  :dribbble
  :reddit
  //... etc

Final icing on the cake — in each service, the only public element is the Dagger module class itself. Everything else (including its inner methods, the service implementation, etc) is Kotlin’s internal visibility, which really drives home the isolation and low API surface area. The integration point literally can’t know anything about a service’s internal APIs — the compiler won’t let it!

Conclusion

I’m pretty happy with how things ended up. Once the Service architecture became concrete, things really fell into place quickly. I was able to migrate all the services over the course of about two days with relative ease, tweaking some things here and there as they came up. Some other benefits along the way were that it becomes trivial to add new services to it, and it’s easy to test (if I wrote them… #sideprojecting).

Future

Is this good now? I think so. Is it done? Definitely not. There’s a few things next on the horizon with this architecture

  1. Manually having to wire up each downstream module to the integration point is a bit annoying and tedious. I have some ideas of how to automate this with code gen that I hope to get done in the next couple weeks.
  2. The Service API can be opened to be more flexible and add support for new features. One such feature I plan to add next is the ability for services to specify their own preferences file in xml resources via field in ServiceMeta. CatchUp itself could just read this off and show the inflated xml in a generic preferences container for that key in settings, abstracted away from the content details.
  3. Having the UI so decoupled from implementation details means that prototyping different UI patterns now becomes very easy. Two things I want to try next are some sort of tablet UI story and Litho.
  4. I think this same system could be leveraged to allow services to provide their own debugging tools, such as mock responses/data.

Further reading:


Thanks to Ron Shapiro and Florina Muntenescu for reviewing this post.


Note: this was originally posted on my Medium account, but I have migrated away from Medium to this personal blog.