![\"Paging](\"/static/paging_demo-a325402259bf9864dced311c9eafa7c8.gif\")
The AUTO1 microservice landscape is growing in size day by day, with new services from a spectrum of languages being erected and usually communicating with each other by using REST contracts.
\nIn a fast paced environment, such as AUTO1, often the service providing the REST API and the service consuming this REST API are developed in parallel, by independent teams. On top of that the business requirements over the lifetime of a project evolve, thus impacting the stability of the API. Having these moving parts in picture can make the development process quite challenging, especially when aiming for high productivity, zero downtime and independent development of the client and server parts. Hence, having a clearly documented and language agnostic API is vital for the success of the whole process.
\nAfter extensive research, we found a great candidate seemingly accommodating all the aforementioned requirements - OpenAPI. In short the OpenAPI specification defines a language agnostic standard allowing to describe REST APIs in a human readable form, imposing a unified structure, clearly and comprehensively aligning expectations between the consumer and provider sides.
\nBelow is an example of such an API described using the OpenAPI 3.0 standard:
\nIn general, we distinguish two approaches when developing APIs. The code first approach, which is basically about developing the source code fulfilling the business requirements and afterwards, optionally, generating the API specification from it, which for OpenAPI can be done using the swagger annotations. The second approach known as design first, is all about describing the API in a human readable language first, by using for example the OpenAPI specification, and then optionally, based on this artifact generating the source code for server and/or client. Naturally, both approaches come with pros and cons.
\nAs previously mentioned, in AUTO1 it often happens that changes that are being introduced in the API on the server side need to be implemented at more or less the same time on the client side. In such a situation, if the code first approach would be practiced, the consumers would need to wait for the server side implementation to be delivered before starting their part, that would be simply inefficient. Here, for the rescue comes the design first approach, where from the moment when the API specification is agreed upon, both server and consumer side can kick-off their side of the implementation, independently. Additionally, this allows to spot misalignments already at a very early stage, and rectify them before the actual implementation from either of the sides is started, hence less expensive to fix.
\nThe OpenAPI comes with a rich ecosystem, from UI tools such as Swagger UI allowing to visualize and test ad-hoc the specifications to AWS integration providing a fast lane to expose the specification via AWS API Gateway.
\nAnother very helpful set of tools are the code generators, that can generate client side libraries and server side stubs from the OpenAPI definitions. These extendible generators support a variety of languages and frameworks, which lets one move faster, especially in a polyglot microservice environment. For our Spring Java applications we have decided to go for a generator that is very well configurable and easily customizable. It can be executed in various ways such as via cli, maven/gradle plugin, etc. Since our Java projects use maven for the build automation process, the maven openapi generator plugin was the natural choice for us.
\nDespite the vast configuration possibilities offered by the OpenAPI code generators, in order to integrate OpenAPI into our AUTO1 development world seamlessly and make the generated API and models classes adhere to our coding standards we needed to extend the default Spring generator that we’ve decided to go with in first place, to include some AUTO1 specific code structures.
\nTo be more productive, concise and spare some boilerplate code we use in AUTO1 the Lombok library, a lot, hence we couldn’t afford not having the Lombok annotations included in the generated classes as well. To follow the OpenAPI generator practices, we have added a configuration property to control enabling and disabling the usage of Lombok annotations:
\nThe above setting would result in generating the following annotations in the output classes:
\nSometimes the need arises to define custom annotations but only on some specific classes, for that purpose we have introduced a new attribute to our OpenAPI specifications called x-extra-annotation, that hints the generator to add additional annotations to the generated code. For example:
\nThe above attribute would result in generating the following annotations in the output classes:
\nGenerics is a very powerful feature that is available and widely used in the Java language for almost two decades now. Unfortunately, as of the time of writing of the article the OpenAPI does not support out of the box inheritance together with generics. To make it available to our developers we have introduced yet another attribute in the API specification named x-make-parent-generic specifying the genericType sub-attribute that indicates a generic type of the parent class (in this case PageDTO):
\nThe above attribute would result in generating the following generic parent in the output classes:
\nAs mentioned at the beginning of this article, our Java services are using the Spring Cloud stack with declarative Feign REST clients, originally the Spring Cloud Netflix Feign module was used for that purpose, however, with recent upgrades to newer Spring versions it was replaced by the Spring Cloud OpenFeign module that resulted in changes in the imported packages (import org.springframework.cloud.netflix.feign.* was replaced with import org.springframework.cloud.openfeign.*). So we ended up in a situation where services use either the old Feign clients (located in their respective packages) or the new ones, yet share the same OpenAPI specification. To support both cases we made it configurable for consumers to go with either one:
\nAnother shortcoming of the default OpenAPI Spring MVC generator was the lack of support for making the generated classes implement specific interfaces (using the “implements” keyword), as of version 4.3.1 only the “extends” keyword was supported. We have overcome this limitation by introducing a custom attribute x-codegen-implements allowing us to specify a number of interfaces the generated class should implement.
\nThe generated output for the above configuration can be found below:
\nTeams often run into the situation where they have to use multiple OpenAPI specifications to generate the necessary contracts to communicate with the upstream services. Unfortunately, the org.openapitools:openapi-generator-maven-plugin requires a separate configuration for each specification, which was a bit too verbose and too cumbersome for us.
\nIn order to circumvent this issue we have created our own maven plugin The main aim of it was to reduce the boilerplate configuration, yet keep the flexibility of the original plugin:
\nWhen dealing with a large number of microservices a common case is that they share common features, hence common models, within a single bounded context. In a typical Code First approach operating in the Java ecosystem this problem would be solved by sharing a common jar which would be then included by the interested parties. We were interested in achieving similar functionality but from the OpenAPI perspective. We achieved this by simply extracting part of the common functionality into separate specification files and referencing them from the individual OpenAPI specifications that would like to reuse these definitions.
\nThe challenge in having OpenAPI specifications referencing other OpenAPI specifications was the dependency management and versioning part. It was especially hard from the consumer perspective, since consumers would need to include both specifications in their respective versions (the referencing one and the referenced one) in order for the code generator to succeed. The solution for this was building API bundling, by adding to the specification a meta attribute called x-bundle-apis we could make all referenced definitions to be included in a single final artifact that was then uploaded in the CI pipeline to artifactory. This meant, from the consumer perspective, that they just needed to include a single dependency now, making their life significantly easier.
\nBy default, when bundling the API we’ve always included the latest referenced specification. Since we have had cases where we wanted to give more control to the API owners to specify which concrete version of a “common API” they would like to reference, we’ve included another attribute x-bundle-imports which could be used as following:
\nDespite the vast number of improvements we have already made in order to retrofit OpenAPI and its tooling into our existing AUTO1 ecosystem and practices, there is still room for improvements. Just to name a few. Although OpenAPI specification itself is language agnostic, we have language specific generators which in some cases need to be instructed on how to generate the output code, and unfortunately these generator hints often make sense only in context of a specific language. As an example, instructing the generator to make the generated Java classes implement some specific java interface. Since the OpenAPI specifications are enriched directly with these hints, it results in obscuring and cluttering the contract definition.
\nYet another challenge would be to generate more complex classes resembling the structure as seen when written manually in the times from before the OpenAPI approach was adopted.
\nAs presented in the article, OpenAPI is a very powerful tool that enabled our AUTO1 organization to move forward faster in a more structured and unified way in terms of defining and evolving the REST APIs. It’s not a silver bullet, as the design first approach and the OpenAPI specification come with a number of challenges by themselves. However, with the talented engineering force we have we were able to resolve most of them in an efficient manner, yet every organization planning to adopt this tool needs to weigh in the benefits and the shortcomings and make the call for themselves.
","fields":{"slug":"/openapi-journey/","tags":["auto1","engineering","openAPI"]}},"categoryArticles":{"edges":[{"node":{"id":"cb45e6dc-ba61-576d-adc2-ba609b0e722a","frontmatter":{"category":"Engineering","title":"Go 1.23: the new Pattern property in http.Request","date":"2024-11-26","summary":"A short review of the new Pattern property in net/http.Request","thumbnail":null,"authorName":"Andrei Gusev","authorDescription":"Senior Software Engineer at AUTO1 Group","authorAvatar":{"relativePath":"pages/go-123-the-new-pattern-property-in-http-request/avatar_Andrei.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAUBBgT/xAAWAQEBAQAAAAAAAAAAAAAAAAAAAQL/2gAMAwEAAhADEAAAAaeTsi2mDmfWV0wzf//EAB0QAAICAQUAAAAAAAAAAAAAAAECAAMEEBESEyH/2gAIAQEAAQUCtYiUvyE71ZkuQNuzaYvuQTP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/AR//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAcEAACAgIDAAAAAAAAAAAAAAABEQACEBIhMTL/2gAIAQEABj8C4KnprGnRhqdnHWpWKPH/xAAdEAACAgMAAwAAAAAAAAAAAAABEQAhEEFRMXGB/9oACAEBAAE/IRoFjyQHGILdksEYAYLXeSuANz4mvAhI736hHP/aAAwDAQACAAMAAAAQNBc+/8QAFhEAAwAAAAAAAAAAAAAAAAAAABAR/9oACAEDAQE/ECv/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAcEAEBAAMBAAMAAAAAAAAAAAABEQAhUTEQYaH/2gAIAQEAAT8Q0H5agDhjJzCu2+nxvKetJ9/fd9x0k6GIzjdTPf69KX9xcAAKlBR2wEDDmf/Z","width":50,"height":50,"src":"/static/479e481648d69b84a342bd2575a225d4/d2d31/avatar_Andrei.jpg","srcSet":"/static/479e481648d69b84a342bd2575a225d4/d2d31/avatar_Andrei.jpg 1x,\n/static/479e481648d69b84a342bd2575a225d4/0b804/avatar_Andrei.jpg 1.5x,\n/static/479e481648d69b84a342bd2575a225d4/753c3/avatar_Andrei.jpg 2x,\n/static/479e481648d69b84a342bd2575a225d4/31ca8/avatar_Andrei.jpg 3x"}}},"headerImage":null},"html":"In Go 1.23, a new Pattern property in net/http.Request was added.\nIt contains the route pattern used to handle the request. This improvement may have gone unnoticed by many, but it can\nsimplify request processing and improve performance in cases where we need to obtain the matched routing pattern\nfor the current request. Let’s see how we can use this feature.
At AUTO1, we have more than 200 microservices, some of which are written in Go and include an HTTP server.\nIt is critically important for us to collect metrics from these services in order to quickly respond to incidents,\nmake data-driven decisions, and optimize system performance.\nFor example, we collect http_requests_total metric:
requestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{\n\t\tName: \"http_requests_total\",\n\t\tHelp: \"total request count\",\n\t}, []string{\"path\", \"method\", \"code\"})Obviously, for the path label, we should store the path template, not the specific value, for example, /test/{id} instead of /test/123.\nOtherwise, we risk a cardinality explosion, which can significantly increase the amount of stored data. You can read how to choose labels properly in \"Prometheus best practices\".
Previously, to extract the route pattern, we had to pass the multiplexer to a middleware function:
\nfunc metricsMiddleware(next http.Handler, mux *http.ServeMux) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\t_, pattern := mux.Handler(r)\n\t\t\n\t\t...\n\t\trequestsTotal.WithLabelValues(pattern, method, statusCode).Inc()\n\t}\n}In this approach, we matched the request to the route a second time, and that negatively affected performance.
\nStarting with Go 1.23, we can get the route pattern directly from the request, without passing the multiplexer to the middleware:
\nfunc middleware(f http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tpattern := r.Pattern\n\t\t\n\t\t...\n\t\trequestsTotal.WithLabelValues(pattern, method, statusCode).Inc()\n\t})\n}This significantly simplifies the middleware, making the code cleaner and more readable. We are no longer dependent on ServeMux and no longer need to re-match the request.
It is important to remember that Request.Pattern is only available after the request has been matched to a route. For example:
func main() {\n\tmux := http.NewServeMux()\n\tmux.Handle(\"/test/{id}\", routeMiddleware(handler()))\n\n\tsrv := http.Server{\n\t\tAddr: \":8080\",\n\t\tHandler: globalMiddleware(mux),\n\t}\n\n\tif err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {\n\t\tlog.Fatalf(\"server error: %s\", err)\n\t}\n}\n\nfunc routeMiddleware(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tlog.Printf(\"ROUTE pattern: %s\", r.Pattern)\n\t\th.ServeHTTP(w, r)\n\t})\n}\n\nfunc globalMiddleware(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tlog.Printf(\"GLOBAL pattern: %s\", r.Pattern)\n\t\th.ServeHTTP(w, r)\n\t\tlog.Printf(\"GLOBAL pattern after: %s\", r.Pattern)\n\t})\n}If we send a request GET http://localhost:8080/test/123, in logs we will see:
2024/11/10 13:20:37 GLOBAL pattern: \n2024/11/10 13:20:37 ROUTE pattern: /test/{id}\n2024/11/10 13:20:37 GLOBAL pattern after: /test/{id}This is the expected behavior. The multiplexer implements the http.Handler interface, and inside the ServeHTTP method, it searches for the handler that matches the request. Starting from Go 1.23, it fills the Pattern property of the request with the matching route pattern:
// net/http/server.go\nfunc (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request) {\n\t...\n\tvar h Handler\n\tif use121 {\n\t\th, _ = mux.mux121.findHandler(r)\n\t} else {\n\t\th, r.Pattern, r.pat, r.matches = mux.findHandler(r)\n\t}\n\th.ServeHTTP(w, r)\n}Thus, the r.Pattern is filled only after the ServeHTTP method of the multiplexer has been called, and the logs confirm this behavior.
In the last two Go releases, the net/http package has received significant improvements that reduced the gap with third-party packages, like gorilla/mux. Now, net/http.ServeMux supports HTTP methods and wildcards in routing patterns, bringing its functionality closer to what third-party packages previously provided.
Additionally, the introduction of the net/http.Request.Pattern property in Go 1.23 has simplified middleware development by allowing us to access the route pattern without needing to re-match the request, improving performance and reducing code complexity.
With each of these improvements, there are fewer reasons to choose third-party routers. While the choice between the standard package and third-party routers used to be more obvious, today I would recommend starting with the standard net/http package.
Paginated APIs, often encountered when dealing with servers and databases, break down large datasets into manageable chunks or pages. This methodology not only optimizes resource usage but also enhances the overall user experience by delivering content progressively. However, integrating and managing paginated data in a mobile app can be a challenge, from maintaining loading states to ensuring a seamless transition between pages and supporting content filtering.
\nIn this guide I will show you how to work with paginated APIs in Android with Jetpack Compose and Paging 3 libraries.
\n
\n![\"Paging](\"https://developer.android.com/static/topic/libraries/architecture/images/paging3-library-architecture.svg\")
The key components of the Paging 3 architecture include
\nWe start with the data layer and define a Retrofit API for fetching movies from network service.
\ninterface MoviesApi {\n @GET(\"/3/discover/movie?language=en-US&sort_by=popularity.desc\")\n suspend fun discover(\n @Query(\"api_key\") api_key: String,\n @Query(\"page\") page: Int,\n @Query(\"with_genres\") genres: String,\n ): Response<MoviesNetworkResponse>\n}Next, we will leverage the Room library, which is compatible with Paging 3, to create a local paging source.
\n@Dao\ninterface MoviesDao {\n @Query(\"SELECT * FROM movies ORDER BY id ASC\")\n fun getMovies(): DataSource.Factory<Int, MovieDbEntity>\n}It's important to highlight that we utilize DataSource.Factory as the return type instead of PagingSource. This choice is made specifically because DataSource.Factory offers the map() method, enabling us to map MovieDbEntity instances into domain layer models.
Now we need to put together local and remote data sources in Remote mediator.
\ngetRemoteKey().nextPage) is stored in the database after each successful network request.val page = when (loadType) {\n LoadType.REFRESH -> 1\n LoadType.PREPEND -> null\n LoadType.APPEND -> repository.getRemoteKey()?.nextPage\n } ?: return Success(endOfPaginationReached = true)For bidirectional pagination support, provide the preceding page number for the PREPEND load type. The REFRESH load type is employed when content is refreshed, initiating loading from the initial page.
\nCall our network data source method to fetch a page of data from the server.
\nval movies = moviesDataSource.discover(page)The RemoteMediator code will appear as follows:
\nclass MoviesRemoteMediator(\n private val repository: MoviesRepository,\n private val moviesDataSource: NetworkMoviesDataSource,\n) : RemoteMediator<Int, Movie>() {\n\n override suspend fun load(loadType: LoadType, state: PagingState<Int, Movie>): MediatorResult {\n\n return try {\n\n val page = when (loadType) {\n LoadType.REFRESH -> 1\n LoadType.PREPEND -> null\n LoadType.APPEND -> repository.getRemoteKey()?.nextPage\n } ?: return Success(endOfPaginationReached = true)\n\n val movies = moviesDataSource.getMovies(page)\n\n val nextPage = if (movies.isNotEmpty()) page + 1 else null\n\n repository.insertMovies(movies, nextPage, loadType == LoadType.REFRESH)\n\n Success(endOfPaginationReached = movies.isEmpty())\n\n } catch (e: Exception) {\n MediatorResult.Error(e)\n }\n }\n}RemoteMediator uses the repository to store movies list and next page number in the database.\nRepository also takes care of clearing the database table when data is refreshed:
\nclass MoviesRepositoryImpl(\n private val database: MoviesDatabase\n): MoviesRepository {\n\n override fun getMovies() = database.movies().getMovies().map { it.toDomain() }\n\n override suspend fun insertMovies(movies: List<Movie>, nextPage: Int?, clear: Boolean){\n\n database.withTransaction {\n if (clear) {\n database.movies().clear()\n }\n\n database.movies().putMovies(movies.map { it.toDbEntity() })\n\n if (nextPage != null) {\n database.remoteKey().insertKey(RemoteKey(MoviesDatabase.MOVIES_REMOTE_KEY, nextPage))\n }\n }\n }\n}The Pager object serves as a bridge between remote and local data sources. It is managing the page loading, prefetching, and handling the transitions between various data sources to provide a seamless and responsive flow of paginated content within your application. You can configure the page loading behavior with the following parameters:
\nclass MoviesScreenViewModel(\n moviesSource: MoviesRemoteMediator,\n repository: MoviesRepository,\n private val movieDetails: MovieDetailsApi,\n) : ViewModel() {\n val moviesFlow = Pager(\n config = PagingConfig(\n pageSize = PAGE_SIZE,\n prefetchDistance = PREFETCH_DISTANCE,\n initialLoadSize = INITIAL_LOAD,\n ),\n remoteMediator = moviesSource,\n pagingSourceFactory = repository.getMovies().asPagingSourceFactory()\n ).flow.cachedIn(viewModelScope)\n}Now, let's explore how to consume the paginated data in your Compose UI.
\nUse the collectAsLazyPagingItems extension function to collect the PagingData and observe changes.
val movies = viewModel.moviesFlow.collectAsLazyPagingItems()Use the LazyColumn or LazyVerticalGrid to efficiently display paginated data in your Compose UI.
LazyVerticalGrid(\n modifier = Modifier\n .fillMaxSize()\n .padding(16.dp),\n columns = GridCells.Fixed(3),\n horizontalArrangement = Arrangement.spacedBy(16.dp),\n verticalArrangement = Arrangement.spacedBy(16.dp),\n ) {\n\n items(\n count = movies.itemCount,\n key = movies.itemKey { it.id },\n ) { index ->\n\n movies[index]?.let { movie ->\n MovieCardSmall(\n movie = movie,\n onClick = { onClick(movie.id) },\n )\n }\n }\n }\n\nNote the usage of the key property for items. The key property uniquely identifies each item, allowing Compose to efficiently update and recycle composables as data changes.
\n
LazyPagingItems offers insights into the ongoing data loading process, allowing you to effectively manage errors during pagination. By examining the loadState.append property, you can determine the loading state when appending new items and present a user-friendly indication of the current status.
\nFor instance, the following code snippet demonstrates how to react to different loading states during item appending:
\nwhen (movies.loadState.append) {\n is LoadState.Loading -> { ProgressFooter() }\n is LoadState.Error -> { ErrorFooter(onRetryClick = { movies.retry() } }\n}Similarly, you can assess the loading status when the list data is refreshed by checking the loadState.refresh property. Extension functions isError() and isLoading() provide convenient checks for error and loading states, respectively.
\nfun LazyPagingItems<Movie>.isError(): Boolean = \n loadState.refresh is LoadState.Error && itemSnapshotList.isEmpty()\n\nfun LazyPagingItems<Movie>.isLoading(): Boolean = \n loadState.refresh is LoadState.Loading && itemSnapshotList.isEmpty()Incorporate these checks into your UI logic to display appropriate screens based on the loading or error states. The following code snippet demonstrates how to handle loading and error states along with providing a retry option:
\nwhen {\n movies.isLoading() -> LoadingScreen()\n movies.isError() -> ErrorScreen { movies.retry() }\n else -> MoviesGrid(movies = movies, onClick = viewModel::onMovieClick)\n}Additionally, LazyPagingItems offers a convenient retry() method, allowing you to retry the last failed data loading attempt with ease.
Now lets add filters to our movie list to see paginated list of movies filtered by genre.
\n![\"Filters](\"/static/paging_demo_filters-ff80a7606fb30961e2bda56bd6ed9985.gif\")
We start with requesting genres from the server and saving them in our database. We will keep genre data and selection in the database so that we can observe it from our domain and UI layers.
\ninterface MoviesApi {\n @GET(\"/3/genre/movie/list?language=en-US\")\n suspend fun genres(@Query(\"api_key\") api_key: String): Response<GenresNetworkResponse>\n}\n\n@Dao\ninterface GenresDao {\n\n @Insert(onConflict = OnConflictStrategy.REPLACE)\n suspend fun putGenres(genres: List<GenreDbEntity>)\n\n @Upsert(entity = GenreDbEntity::class)\n suspend fun updateGenres(genres: List<GenreDbUpdateEntity>)\n\n @Query(\"SELECT * FROM genres\")\n suspend fun getGenres(): List<GenreDbEntity>\n\n @Query(\"SELECT * FROM genres\")\n fun observeGenres(): Flow<List<GenreDbEntity>>\n}Now we can add with_genres parameter to our network request:
@GET(\"/3/discover/movie?language=en-US&sort_by=popularity.desc\")\n suspend fun discover(\n @Query(\"page\") page: Int,\n @Query(\"with_genres\") genres: String,\n @Query(\"api_key\") api_key: String,\n ): Response<MoviesNetworkResponse>Remote mediator will get selected genres from the repository
\n ...\n val genres = repository.getGenreSelection()\n val movies = moviesDataSource.getMovies(page, genres)\n ...Every time user selects a new filter we stave it in the database and restart pagination by clearing movies in the database and resetting the page to 1.
\nclass MoviesScreenViewModel {\n\n fun onGenreClick(genreItem: GenreItem) = viewModelScope.launch {\n genresProvider.setSelection(genreItem.id, !genreItem.isSelected)\n repository.clearMovies()\n }\n \n}class MoviesRepositoryImpl {\n\n override suspend fun clearMovies() = withContext(Dispatchers.IO) {\n database.withTransaction {\n database.movies().clear()\n database.remoteKey().insertKey(RemoteKey(MoviesDatabase.MOVIES_REMOTE_KEY, 1))\n }\n }\n\n}
Source code: https://github.com/auto1-oss/android-compose-paging\n
Official documentation: https://developer.android.com/topic/libraries/architecture/paging/v3-overview
Sense of Security
\nTreat the injured first
\nTrust your tools
\nCommunication via Radio
\nWhat I present today is how to use police techniques and their mindset to detect and solve bugs in software, so that you can be the best software developer that society can provide.
\nFor some years, I worked as a police officer in Brazil and I can say that it is a challenging career. The first step was to pass a national test, where knowledge of various laws was required. Criminal laws, traffic laws, and even the laws of physics.
\nThe second step was to pass a battery of medical exams. Almost perfect health was necessary to be able to practice the profession, in addition to having most of the parts of the human body.
\nThe third step was the psychological examination. It was necessary to have a clear mind and not have impulses different from those expected from a person who must protect other people.
\nAs a fourth part, there was also the fitness test, a test in which career candidates had to prove they had sufficient strength and speed to carry out daily police tasks. Running, pull-ups, sit-ups and long jump are examples of what was required.
\nThe last stage was the Police Academy, where for three months those from now on known as students would learn and prove the learning of several new skills: first aid, use of firearms (pistols, carbines, submachine guns, etc.), non-lethal weapons, martial arts, public policy, and many other disciplines.
\nAll this so that society receives the best police officer that its people can provide. So, what about using some of the following principles and techniques to become the best software developer that society can provide?
\nWithout needing to be a security expert, you already know that it is impossible to have a completely safe city. Even if we reached the point of having perfect police officers, which in itself is impossible, we would not have a police officer for every citizen. In general, the recommendation is 300 police officers for every 100 thousand inhabitants.
\nAware of this limitation, that it is not possible to guarantee complete security for all citizens, police agencies work with the concept of a sense of security. Through strategic actions to combat crime, citizens' feeling that they are safe is increased. This mental condition gives them peace of mind when working, studying, and carrying out their daily tasks.
\nThis sense of security also affects criminals. The common visibility of police officers carrying out their duties in strategic places, as well as effective actions to combat crime, make criminals stay away from that place.
\nNotice that ending 100% of crime is impossible, what we do well is limit its actions. From here we can extract our first lesson when dealing with software: perfect code, totally bug-free, is impossible.
\nDespite this conclusion, the fact that it is impossible does not mean that we should simply ignore the problem. On the contrary, we can, like the police, create strategic actions to combat Bugs. To the point that Bugs themselves will run away from our code.
\nTest Driven Development - TDD is a strategic action that provides a feeling of security in your code. Before the crime occurs, you position police officers in a specific region. Before the Bug occurs, you position the test in a specific place in your code.
\nRobert C. Martin, in his book \"Clean Code: A HandBook of Agile Software Craftsmanship\", teaches us that writing unit tests before writing production code is just the tip of the iceberg in defining TDD. There must be a continuous cycle of improvement, in which there is always a test covering the code you are developing and this security that the code is protected allows you to play with it, and work with it. After all, whatever problem may happen, there is a police-test there to protect you.
\nOne of the functions of Highway Police Officers is to provide first aid in the event of traffic accidents. There are often serious injuries that must be treated as urgently as possible, as even 1 minute can make the difference between life and death.
\nThe first step when arriving at a traffic accident site is to mark the location appropriately to prevent other accidents. We must use lighting devices and other appliances to make it clear to other vehicles that there is a danger there.
\nThe police officer must then check the condition of all the injured to prioritize them according to the severity of each situation.
\nOnce the treatment of each injured person has been completed and, if necessary, they have been taken to the hospital, it is time to survey the accident site. At this stage, the police officer begins to collect all the information relating to the accident to have sufficient information so that a decision can be made on the cause of the accident and, when applicable, any culprits.
\nThis is the mental model that you and I must adopt when we are faced with a Bug. Although several procedures have been adopted to prevent it from occurring, it is a fatality that can happen. It's like when the highway was well designed, the vehicles were up to date with maintenance, but even so, due to carelessness or misunderstanding on the part of the driver, an accident happens.
\nAs in the case of traffic accidents, the first step to take is to secure the environment so that the Bug does not cause other problems. If possible, we can revert the application version to one in which the Bug did not appear. We can use a feature flag to temporarily disable a feature. Regardless of the method used to stop the bleeding, the important thing is that we must focus on treating the injured first rather than looking for culprits.
\nWe have to agree with David Thomas and Andrew Hunt, in their work \"The Pragmatic Programmer: your journey to mastery\" that facing Bugs is a sensitive subject for most developers. Instead of treating the issue as a problem to be solved, some approach it with denial, pointing fingers, lame excuses, or simply apathy. It's exactly the opposite of what is expected!
\nInstead of looking for the culprit in a traffic accident, we must first assist the victims and stop the bleeding. In software, instead of finding out who inserted the Bug, we must solve it.
\nWhen treating an injured person in an emergency, one approach is to use the Glasgow Scale to determine the patient's consciousness. On this scale, one of the components is the verbal response.
\nThe injured person is asked simple questions, and by analyzing the responses, we can assess the level of consciousness and consequently the level of urgency of their care.
\nYou and I must carefully read every error message that is displayed during the Bug, it may contain vital information so that we can resolve the problem.
\nYou may be thinking, but what if there is no error message? If we are just faced with an incorrect result? Well, this also happens in traffic accidents, it is not uncommon for there to be an unconscious victim who cannot answer questions.
\nWhen we discussed the sense of security, we talked about the importance of tests in our application. Go and build a test that can reproduce that Bug, so you can extract the information you need to solve it.
\nThink of these tests like the motor tests that highway patrol officers perform on unconscious victims to check their level of consciousness: do the eyes open in response to pain? Is there a pupillary response to light stimulation? Mentally translate this into software language: Does the Bug repeat itself when I enter this and that parameter? And if I insert this new argument, does it happen too?
\nTo decide on a Bug, data is needed. If you don't receive this data, you must extract it.
\nPolice work is very complex and your day-to-day job involves the fight for life and protection of the most important assets for society. There would be no point in having a modern and efficient selection and training process for the development of police officers if they were not given the appropriate tools for their work.
\nAnd when I say appropriate, I don't mean they are minimally capable of performing the task, as the result delivered is directly related to the quality and safety of the tools. For an excellent result, not only the police officers but their tools must also be excellent.
\nIn the police agency I worked for, one of the work tools used was an Austrian pistol known worldwide for its safety and efficiency. The Glock 17 pistol was developed to meet the strict criteria of the Austrian army who wanted to replace the model used during the Second World War.
\nIn addition to requirements on durability, efficiency, and reliability, safety against accidental firing was one of the pivots of the new model.
\nBut how did a knife factory manage to deliver the pistol model that would prove to be the safest? Well, it can be said that, in addition to intense testing, the secret to reliability was simplicity and maintainability. The person responsible for the project designed the equipment with as few parts as possible, minimizing complexity. Today the Glock pistol is made with an average of 35 parts, which is a much smaller quantity than other pistols on the market.
\nWith a simpler design, so-called first-level maintenance is much easier. Equipment users can easily perform maintenance procedures, making the equipment's efficiency last for decades.
\nOf course, you must already remember the principle with the acronym KISS, which stands for Keep it Simple (omitting the last word because I don't want to offend the reader). The secret of that renowned pistol model was not in the used materials, but in being simple. The secret of maintaining your software in good shape is to keep it simple, having a specific component for each task. When a problem arises, you will know exactly which component to look for, as each component has a responsibility, this is the principle of Single Responsibility.
\nBecause of all this, when you are at a shooting range practicing your aim, the police officer knows that if he is not hitting the target, most likely, the error is in his procedure and not in the equipment.
\nAllan Kelly, in one of his contributions to the Book \"97 Things Every Programmer Should Know\", corroborates this idea that we should trust our tools when dealing with Bugs. Considering our tools are widely used, mature, and in use by multiple companies, we have little reason to doubt their quality.
\nOf course, a prototype, version 0.1, is much more fragile than a library that has been on the market for several years. We are also not saying that it is impossible to have bugs in compilers.
\nBut considering how rare compiler bugs are, what we are saying is that in the search for a Bug solution, we should trust our tools and look at our piece of code first.
\nImagine it as an efficient distribution of efforts. The library or compiler code has already been reviewed and tested for many years by countless people. As for that little piece of code of yours, only God knows who besides you has looked at it on your team.
\nOn one of those days, I managed to solve an unpleasant bug that occurred in one of our services. It was a non-deterministic error in our test suite, that is, the Bug only pushed out its head when no one was looking.
\nThis type of non-deterministic error generally involves three types of origin: multithread, time sensitivity, and dissynchrony between construction and cleaning of test data.
\nAnalyzing the body of the failed test, after several hours, of course, I began to strongly distrust the way we created timestamps with a native Java library. After all, the test seemed perfect to me, if there was an error it must have been somewhere else.
\nBut, aware of this lesson that we should trust our tools, especially since the component in question had been in Java for 9 years, I ignored this distrust. This saved me a lot of time and embarrassment, as I'm going to share that the error was elsewhere.
\nOur test suite ran all tests in a single thread, so it wouldn't be a concurrency issue. But it also ran each test in a deterministic but unpredictable order. Therefore, the source of the intermittent problem must be in some specific order in which the tests were performed.
\nAfter several hours of analyzing the logs, I was able to identify two tests that failed when run in sequence. It turned out that, although our tests performed a test data cleaning routine after each execution, a simple typo had silently overwritten the cleaning method in one of the tests, which generated data inconsistency if such tests were executed in a specific order.
\nI remember that one of the first systems I developed when I worked in the police was one to replace the sending of statistics through radio communication. Many years ago, each police station was supposed to send daily work statistics via radio. Data such as number of people and vehicles inspected, car accidents, etc.
\nThis took a lot of time as the audio quality of radio communications was not always ideal. The interlocutors spent a lot of time transmitting and confirming each piece of information. All this using Q-Code
\nThe Q-Code is a standardized collection of three-letter codes used in radio communication. The use of Q-code even in voice transmissions makes communication safer and more efficient. Each set of three letters, always starting with the letter Q, has a specific meaning that is known to the interlocutors.
\nI could see that two of those codes were widely used among police officers in radio communication: QAP and QSL.
\nQAP was used when you wanted to transmit a message. Let's say someone wanted to talk to me, so he said on the radio: \"Bruno, QAP?\". This could be translated as \"Bruno, are you ready to hear a message?\"
\nMy answer could be QAP or for example QRM. If I responded with QAP, I was signaling that yes, I was ready to receive the message. If you responded with QRM, I would be informing you that at that moment the communication is being interfered with.
\nDuring communication, the second most used Q-Code appears, QSL. At the end of each message, the sender ends it with the QSL. This means that he is asking the receiver to confirm whether they received the message. It would be something like: \"There is a suspicious vehicle heading towards your team, QSL?\" If I had understood the entire message, I should respond with a simple \"QSL\". In the absence of a response, the sender repeats the message until receiving confirmation from the receiver.
\nEach message is concluded with a QSL, and its reception is confirmed with a QSL. Does this remind you of any transmission control protocol out here, used on the internet?
\nThe main thing here is that a large amount of knowledge is condensed into three letters, in a language that provides interlocutors with efficiency and security. Eric Evans, the author of \"Domain-Driven Design: Tackling Complexity in the Heart of Software\", would agree with me that there is a principle here that should also be adopted in software development.
\nA language specific to the domain, which considers possible interference from the environment, a ubiquitous language that is known by all interlocutors, can be the difference between life and death, between success and damnation of a project.
\nWhen a project does not have its own language for that domain, this creates the need for numerous translations between interlocutors. Developers have to translate for domain experts. These, in turn, have to translate their demands to developers. Even among developers, there may be a need for translations.
\nAll these translations make concepts confusing and the consequences of a lack of mutual understanding can be disastrous.
\nThe importance of this in trying to arrest a Bug goes much further than compilation errors or exceptions. It goes through the swampy characteristic that the software may be logically working almost perfectly, but it is not delivering what was expected by the stakeholder.
\nLet's look at the following pieces of code:
\nif (order.status != completed) {\n // (...)\n}if (order.isFullyRefunded()) {\n // (...)\n}Here we have a small example of two ways of expressing the same situation. In the second piece of code, a common language between developers and domain experts was used. Here, possible logical errors can be easily noticed, without the need to keep in mind what the possible states of the Order are and what the expected state is for that case.
\nSee the following code:
\nPayment newPayment = payment.add(10);What do you expect from this code? Does it add ten dollars to the payment? Ten transactions, inbound or outbound? Will the method create a new payment object or will it just change the value of the payment variable and create a new reference called newPayment?
\nA lesson also taken from the book \"Clean Code: A HandBook of Agile Software Craftsmanship\" shows that clear communication must also take place between software developers. Functions must communicate exactly what they are doing.
\nIf you have to look at the function's implementation to know what it's actually doing, that's a big sign that you need to rename it.
\nI'm very grateful that you've made it this far, you've certainly learned a lot about police activity and how some of its principles can be applied to combat and remove Bugs from your software.
\nBy applying this knowledge you will be able to work more confidently in your code, giving you peace in your daily life and also allowing you to experiment and discover new things.
\nYour aim will be improved and you will get straight to the point, avoiding damage instead of first looking for the culprits. You become bolder and go further because you know you can trust your tools. This leaves you more time to refine your code and seek excellence.
\nFinally, not only will your code be safe and efficient, but its result will also be in line with what your customers expect because through effective communication you can accurately deliver what was demanded from you.
\nThe result of all this is that Bug is in jail, lives are no longer in danger, and you can go home at the end of your shift to enjoy the rest of the day with your family which is looking forward to welcoming their hero back.
","fields":{"slug":"/bug-police-officer/","tags":["debugging","testing","TDD","DDD"]}}}]},"authorArticles":{"totalCount":2,"edges":[{"node":{"id":"4762bbde-b461-512b-93ae-22335b823a90","frontmatter":{"category":"Architecture","title":"Harmonizing Application Cockpit data using DB views","date":"2023-02-16","summary":"Our journey building the AUTO1 service inventory search using DB views","thumbnail":null,"authorName":"Mariusz Sondecki","authorDescription":"Mariusz is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/harmonizing-application-cockpit-data-using-db-views/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAWABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAMEBQH/xAAXAQEBAQEAAAAAAAAAAAAAAAACAwAB/9oADAMBAAIQAxAAAAGv2Svu3GUctOAU5wh//8QAGxAAAQUBAQAAAAAAAAAAAAAAAQACERIiEyH/2gAIAQEAAQUCJkDB7KBW+bEJwkNZ6v/EABgRAQADAQAAAAAAAAAAAAAAAAEAAhEQ/9oACAEDAQE/AQMmMLPP/8QAGBEBAAMBAAAAAAAAAAAAAAAAAQAQESH/2gAIAQIBAT8BV2cmFf/EAB4QAAEEAQUAAAAAAAAAAAAAAAABAhEhQRAxUWFx/9oACAEBAAY/AqwSjpMj+oKvRU5PDZD/xAAdEAACAgIDAQAAAAAAAAAAAAABEQAhQWExcZFR/9oACAEBAAE/IWdbBmVNkJTg+IpKCEBHylkKGgktmegIYKT9dwgOydif/9oADAMBAAIAAwAAABAA18H/xAAXEQEBAQEAAAAAAAAAAAAAAAABEQAQ/9oACAEDAQE/EES8YEurv//EABgRAAIDAAAAAAAAAAAAAAAAAAEQESEx/9oACAECAQE/EDJJlS//xAAdEAEAAgICAwAAAAAAAAAAAAABABEhQVFhMZGx/9oACAEBAAE/ECQu0QVxCvgawa6iFVCFbUJDi/sW7HoRRveCLN+4F4gKruoOH4KtotWHILP/2Q==","width":50,"height":50,"src":"/static/41d7e93878c0ba2eb12594d317c2b361/d2d31/avatar.jpg","srcSet":"/static/41d7e93878c0ba2eb12594d317c2b361/d2d31/avatar.jpg 1x,\n/static/41d7e93878c0ba2eb12594d317c2b361/0b804/avatar.jpg 1.5x,\n/static/41d7e93878c0ba2eb12594d317c2b361/753c3/avatar.jpg 2x,\n/static/41d7e93878c0ba2eb12594d317c2b361/31ca8/avatar.jpg 3x"}}},"headerImage":null},"html":"As you may have read about Application Cockpit in one of our previous articles, it is “The tool which lets concerned developers quickly find their bearings and gather relevant information pertaining to a particular service running in our ecosystem”.
\nOver time, AUTO1 engineers and managers grew fond of using Application Cockpit, and started relying on it on a daily basis. The increase in popularity, naturally, led to development of new features, out of which, one of the arguably most impactful ones was the \"dynamic service search\" (or \"App Cockpit Search\", as we call it internally).
\nAUTO1 platform currently consists of 600+ microservices, and on average two new services are created every week. Many times in the past, we needed to search our AUTO1 service inventory for services which were matching some specific criteria, e.g. are written in a specific language, using certain frameworks or libraries, or have certain features enabled.
\nAs it happens, this is actually where App Cockpit App Cockpit Search chimes in. From the user’s perspective, it is an UI that lets the user input some search criteria using the Cockpit Query Language (or CQL for short) and be presented with a list of services matching the criteria. One typical use case is searching the AUTO1 service inventory for the usage of a specific library in a specific version.
\nBelow is an example search for services using org.springframework.boot:spring-boot-starter dependency in version 2.3.0 onwards:
Once we were certain of what we wanted to achieve, i.e.: search our AUTO1 service inventory using dynamically defined criteria, we began the efforts of defining the architecture.
\nAs you may recall from our previous article about the Application Cockpit, we use Backstage as our Front-end. Backstage is fetching data from our backend-for-frontend service called “Application Cockpit Data Provider”, which in turn, is backed by a PostgreSQL DB.
\nAs Backstage itself comes with search functionality that could be potentially used for our purposes, we were left with two options, either to reuse the already provided solution and tailor it to our needs or create a solution from scratch.
\nAfter evaluating the Backstage search, which at that time, was still in the early alpha stage, it was clear to us that it would be a bumpy ride. As it is a full-blown search mechanism, or to put it more bluntly, an over-blown mechanism, compared to what we really needed:
\nAs we already use Spring Data JPA as the persistence layer to communicate with PostgreSQL in our data provider service, we have limited our options to two choices: dynamically built queries on top of joined database tables or database views. Either of them seemed to us, initially, as a good candidate for dealing with the complexity that stems from the heterogeneous data structures.
\nThe idea behind this solution was to create a set of SQL queries, backed by corresponding Spring Data JPA repositories, one for each attribute in question, execute these queries against the DB, and then combine the results on the Java side performing the necessary filtering and aggregation.
\nThe advantage of this solution was that we would keep things small and isolated. Each query is in a separate repository. Each one would deal just with the data structures it's aware of. On the other hand, this solution would come with a significant performance penalty, due to the substantial amount of data that would be transferred from the DB, just to be later filtered out on the application side.
\nThe alternative solution would be to dynamically build a large SQL query that would perform both the retrieval of the data and the filtering and aggregation of its results.
\nHowever, this approach would come at an increased maintenance cost. As every data structure change would require a corresponding Java code change, hence we would lose the benefits of having the data values stored in the JSON objects. On top of that, maintaining a large SQL query would quickly become a problem on its own.
\nKnowing the shortcomings of the dynamically built queries, we decided to evaluate another promising approach - the database views, that could hide the complexity of dealing with the heterogeneous data structures:
\nIn that approach, we would create a database view that could look like the following (based on our previous example):
\nwhich returns:
\nThe main database view, here called document_view, would be the aggregation of multiple sub-views. Each sub-view would only know about its data structures, hence we would achieve the desired single responsibility and encapsulation traits, making the solution more extendible and more maintainable. Even if something would be changed within one of the sub-views, it would not affect the aggregated view or other sub-views.
As previously mentioned, since we use Spring Data JPA extensively, the database view would be reflected in the code as a JPA entity which would allow us also to use the Criteria API to build a generic query on top of that main view and utilize the built-in PostgreSQL functions, that would come especially handy when interacting with the JSON objects.
\nAs the DB views seemed like a good solution, we still needed to answer one more question: whether to materialize them or not. As materialized views would be, obviously, faster to query, but that would not come for free, as we would need to build a mechanism to frequently refresh the materialized views, which would also induce a performance overhead. In the end, as we expect the App Cockpit Search to happen fairly infrequently and we value accuracy over query execution time, we’ve decided to not materialize the views.
\nFinally, we decided to move forward with the DB view based solution. Although it’s not a silver bullet and requires us to keep an eye on the query performance, as over time the number of sub-views will grow, which might impact the overall search performance, we consider this to be a fair trade-off for the gained extensibility and maintainability of the search functionality.
","fields":{"slug":"/harmonizing-application-cockpit-data-using-db-views/","tags":["Cockpit","PostgreSQL"]}}},{"node":{"id":"99158851-0237-5904-98ca-02b5e2f45684","frontmatter":{"category":"Coding","title":"Spring 5 candidate component index case study","date":"2019-02-05","summary":"Analysis of Spring 5.X candidate component index applicability to boost our application startup time","thumbnail":{"relativePath":"pages/spring-5-indexer/thumbnailImage.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAZABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAIBBf/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAeBWySoZgAf/xAAaEAACAgMAAAAAAAAAAAAAAAAAARASICEx/9oACAEBAAEFAisLg95//8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAwEBPwEf/8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAgEBPwEf/8QAFBABAAAAAAAAAAAAAAAAAAAAMP/aAAgBAQAGPwJP/8QAGhAAAwADAQAAAAAAAAAAAAAAAAERECAxIf/aAAgBAQABPyFK8Kx5obfaNVLt/9oADAMBAAIAAwAAABBTBzz/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAcEAEAAgMBAQEAAAAAAAAAAAABABEhMUFRYSD/2gAIAQEAAT8QR0FfCY14dprVxKUeSiVurchzX3sU1d7c6ZqWULr2CBDvz9f/2Q==","width":265,"height":325,"src":"/static/85788e7deb4eaccc84801ebd1f7b77a8/a2998/thumbnailImage.jpg","srcSet":"/static/85788e7deb4eaccc84801ebd1f7b77a8/a2998/thumbnailImage.jpg 1x,\n/static/85788e7deb4eaccc84801ebd1f7b77a8/99cce/thumbnailImage.jpg 1.5x"}}},"authorName":"Mariusz Sondecki","authorDescription":"Mariusz is Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/spring-5-indexer/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/spring-5-indexer/headerImage.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwAAAwEAAAAAAAAAAAAAAAAAAAECBv/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAc/UgxB//8QAFBABAAAAAAAAAAAAAAAAAAAAIP/aAAgBAQABBQJf/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPwE//8QAFBABAAAAAAAAAAAAAAAAAAAAIP/aAAgBAQAGPwJf/8QAGBAAAwEBAAAAAAAAAAAAAAAAAAERIEH/2gAIAQEAAT8hElKxzmP/2gAMAwEAAgADAAAAEPPP/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPxA//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPxA//8QAGxAAAgIDAQAAAAAAAAAAAAAAAREAECExQVH/2gAIAQEAAT8Qm5fABBZ6Hh5TKTxX/9k=","width":1280,"height":853,"src":"/static/430c76d0d1e83aad3065e86e11366218/966a5/headerImage.jpg","srcSet":"/static/430c76d0d1e83aad3065e86e11366218/966a5/headerImage.jpg 1x,\n/static/430c76d0d1e83aad3065e86e11366218/aa36d/headerImage.jpg 1.5x"}}}},"html":"Since day one at AUTO1, we strongly believe that the microservice architecture helps us run business at scale not only due to the technical benefits it brings but also, if not foremost by allowing many teams to work independently from each other. Today, we’re running more than 250 microservices in production and on average add one more to our platform every week. All of them are powered by the Spring Framework technology stack, which we value very much for its productivity, but at the same time consider it to be a bit heavy in terms of bootstrapping time. The typical startup time of our Spring applications varies between 40 and 80 seconds. Since we do around 30 deployments per day, our engineering teams spend roughly 30 minutes daily waiting for the deployments to be completed. As we value the time of our engineering teams, we wanted to find a way to decrease these numbers.
\nWhile investigating the startup times of our applications, we have identified classpath component scanning as a potential place to look for improvements. Luckily for us, some of our tech savvy engineers suggested a not so prominent Spring 5.X feature to come at help here - the build time component candidate index.\nThe build time component index is an alternative for regular component scanning, which skips the classpath search operation in favor of using a compile time pre-generated index of component candidates during the spring application context building. However, it comes with a small print from the Spring Framework experts saying that this functionality should have mostly a visible impact on startup times for applications with large amount of beans and that are operating in environments where IO operations are expensive (e.g. remote file systems) or where JVM security managers are in use. Although, neither of this is our use case, we have decided to proceed with the feature evaluation and measurements, as here at AUTO1, we like our decisions to be data driven.
\nThe key part of the indexer feature is a pre-generated index file located in META-INF/spring.components one per JAR. Every index entry in this file is a fully qualified name of a candidate component as a key and comma separated stereotypes as value. So for example “X=Y, Z” can be read simply as register a candidate component X with following stereotypes Y, Z. Below is an example of a spring.components file:
\ncom.auto1.playground.service.SpringDummyService=org.springframework.stereotype.Component\ncom.auto1.playground.domain.SpringDummy=javax.persistence.Entity,javax.persistence.Table,javax.persistence.EntityListeners\ncom.auto1.playground.repository.SpringDummyRepository=org.springframework.stereotype.Component,org.springframework.data.repository.Repository\ncom.auto1.playground.configuration.LocalConfiguration=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomService=com.auto1.playground.service.IndexedCustomService\ncom.auto1.playground.Application=org.springframework.stereotype.Component\ncom.auto1.playground.SpringDummyController=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomExample=com.auto1.playground.annotation.IndexedCustomThe first question that arises here is how to create such a file and keep it up-to-date. The Spring engineers thought about this problem as well and came up with an annotation processor tool called spring indexer that hooks in during the project build phase and generates the file. Making use of it is fairly simple, assuming you are using your favorite build automation tool, in our case maven and want our IDE to keep this file up to date, simply add the spring indexer annotation processor to your pom file:
\n<build>\n <plugins>\n <plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.5</version>\n <configuration>\n <annotationProcessorPaths>\n <path>\n <groupId>org.springframework</groupId>\n <artifactId>spring-context-indexer</artifactId>\n </path>\n </annotationProcessorPaths>\n </configuration>\n </plugin>\n </plugins>\n</build>Once this is done, what happens during project build time is the spring context indexer (precisely speaking the CandidateComponentsIndexer class) while going through the project identifies annotated components and outputs them to META-INF/spring.components. The type of components it includes in the candidate list are:
\nClasses annotated directly or indirectly with spring @Component annotation (which by itself is meta annotated with @Indexed) and other stereotypes for which @Component is a meta annotation such as @Repository, @Controller, @Service, @Configuration.
\nClasses and interfaces annotated with annotations from javax package, including CDI annotations (@Named, @ManagedBean), JPA annotations (@Entity, @EntityListeners) or even Servlet annotations (@WebListener)
\nAny custom classes and interfaces annotated with @Indexed annotation.
\nIt should be noted that applying @Indexed on a class/interface where no other standard stereotype (as from points 1 and 2) is present will not make by itself the bean managed, thus not making them eligible for being autowired, but merely provide a hint that it represents a stereotype to be considered in the index.
\nExample of the @Indexed annotation usage:
\n@Indexed\npublic class IndexedCustomService { }Or as meta annotated as below:
\n@Target(ElementType.TYPE)\n@Retention(RetentionPolicy.RUNTIME)\n@Documented\n@Indexed\npublic @interface IndexedCustom { }\n\n@IndexedCustom\npublic class IndexedCustomExample { }In order to make them available in the application context, thus autowirable, we could for example include them in the component scan filters as shown below:
\n@SpringBootApplication\n@ComponentScan(basePackages= {\"com.auto1\"}, includeFilters = {\n @ComponentScan.Filter(type= ANNOTATION, value = {IndexedCustomExample.class}),\n @ComponentScan.Filter(type= ASSIGNABLE_TYPE, value = {IndexedCustomService.class})\n }\n)\npublic class Application implements WebMvcConfigurer {\n // ...\n}There is one big pitfall that comes with the feature though, that is by default the index is automatically enabled when a META-INF/spring.components file is found on the classpath during application start-up and this in turn, disables completely the regular classpath component scanning. In other words, both the index and classpath scanning cannot be used at the same time. The consequence of this is that if an index is available for some JARs, but is incomplete due to including other JARs that come with potential bean candidates but don’t have the index file generated, those beans will not be discovered, thus application startup might spectacularly fail with a well known error: “X required a bean of type 'Y’ that could not be found”.\nFortunately, Spring engineers, as thoughtful as they are, provided a feature flag spring.index.ignore that can be passed either as a system property or set in the spring.properties file at the root of the classpath allowing us to disable the usage of the candidate component index altogether.
\nLike mentioned before, the aim of our investigation was to understand if the Spring indexer feature is of any help to us in regards to boosting the application start-up times. In order to make the test fairly reliable and control as many confounding factors as possible, we have decided to run the test firstly, using a “dummy” Spring 5 application (in two flavors - with 120 and 5000 beans) using all the features a production microservice would use, on a bare metal machine i.e. a MacBook Pro late 2017 i7. Secondly, running a real life production service in a less controlled but close to production environment i.e. on a r4.8xlarge EC2 instance with other services running concurrently. The applications themselves were using Spring 5.0.10.RELEASE with Spring Cloud Finchley.SR2 release train.\nAfter making a couple of hundred measurements of application bootstrapping times with candidate component index enabled / disabled and verifying in the logs that Spring was indeed using (or not) the component candidate list, we ended up with following averaged results:
\nAs the above results have consistently shown, there was no improvement or even insignificant worsening (which will be a subject of our analysis in a separate article) of the bootstrapping time of the Spring applications, regardless of the number of candidate beans present, when using the pre-generated candidate component index. This was expected, as explained in the opening section, the conditions for which this feature was designed (slow or expensive IO operations) are simply not met in our case. One more important factor to consider here, is that a great deal of the beans in our Spring applications, is created using bean definition factory methods, which are not subject to the component classpath scanning, thus are not impacted by this feature.
\nAll in all, we deemed this feature not to be useful for our purposes and moved on to the next item on our list.
","fields":{"slug":"/spring-5-indexer/","tags":["spring","spring-indexer","bootstrapping","performance"]}}}]}},"pageContext":{"slug":"/openapi-journey/","tags":["auto1","engineering","openAPI"],"category":"Engineering","author":"Mariusz Sondecki","date":"2021-11-22"}}