![](\"https://lh6.googleusercontent.com/-K5rLPtggaNeQpml856NbA9UyUU6_scnBUgUE_W-RA7MVORz_dYAzrQ_jg7a9vBfQJHqmdosoCH7Evd0dbTF8-kzH0dlygQPeoqQu47__E5Pky0GmeyhlcJ1UWNuQrC7j5s3MZvQhj_EoSgwBA\")
When a software-based company grows, it creates lots of new code, which means multiple engineers may be facing the same problems across the whole platform. They will usually come up with various solutions to fix it. Some of them might be generic and reusable, while others will be tailored to one’s needs. As it usually happens, the solutions will then be bundled together to create libraries. This is very good, up to a certain point, since over the years, engineers will create them by the dozens and this, in turn, will bring about other hurdles such as:
\nThis was quite a big pain point for AUTO1 software developers as we have 60+ common libraries that need to work perfectly in 600+ production services across multiple business units and products. One simple mistake, for example, a wrong configuration change leftover after a debug session can end up in disaster.
\nEnabling protection against direct commits for main branches will address the first hurdle - only pull requests with double approvals and tested by CI can be merged.
\nThe next 2 issues are more about ease of usability for the developers, as they might be reluctant to upgrade their service if the process is troublesome. So, in the best-case scenario, our end-user should periodically update only a single property. In addition, our common library developers need to be able to easily publish their changes.
\nThe solution to it was the release trains, usually known as BOMs or for AUTO1 developers, service-commons-dependencies. After successful implementation, you will just need to trigger CI build to get the newly released version, which in turn can be pasted into a pom/gradle file. So, right now everything looks good, but can we be 100% sure that our changes are going to work? Perhaps there is a very small bug in a messaging lib or we will lose tracing for incidents. That’s unacceptable. So let’s ensure that will never happen by introducing automatic real use case testing.
\nIn our approach, we thought most of the problems could be revealed by running extensive integration testing. The test harness probably catches all bugs in our codebase and simply fails when needed preventing the release of a buggy version.
\nBut no matter what you do or how extensive your tests are, you are never going to be able to perfectly mock the production environment.
\nEnter the End to End testing of newly released service-commons-dependencies, more commonly known in AUTO1 from here on out, “E2E testing”.
To get it done, we need CI pipelines that will automatically test our new versions of release trains that are triggered either manually or automatically every week. All tests need to be as close to the real-world scenarios as possible. A practical solution is to create a new service that will include all of the most popular use cases of the aforementioned libs. Then we need to deploy it to all of our environments - as previously mentioned there is always something different between local, QA, and production environments, so we need to cover them all. The tests must be the same for all environments. But the trigger may differ - for example, we are using our test environments to prepare new releases for production.
\nThe following diagram illustrates the basic principle of how it should work:
\nIn our implementation, we chose to use Jenkins with JobDSL pipelines and Groovy for scripting. The first stage of the E2E process is checking whether a new version of service-commons-dependencies exists by using the maven versions plugin:
\nmvn -U versions:update-property -Dproperty='service-commons-dependencies.version'To check for any changes:
\ndef hasChanges = sh(returnStatus: true, script: \"git diff --exit-code\")If nothing new was found let’s just abort the pipeline and wait for another CRON or manual trigger.\nIn case a new release was found, we need to build it, deploy it, and run E2E tests.\nIf there are no failures the pipeline will create a new core-test-service release.\nFor those cases, we are using parameterized jobs, for example, to trigger the E2E tests we can use the following commands:
\nbuild(job: 'java/core-svc/core_test_end_to_end_test', parameters: [booleanParam(name: 'SEND_SLACK_NOTIFICATION', value: \"true\"), string(name: 'RELEASE_TRAIN_VERSION', value: service_commons_dependencies_version)])The new release will be picked up by the production pipeline by checking the latest tag creation time in our repository:
\nString latestTagDateString = sh(returnStdout: true, script: \"git rev-list -1 \\$(git describe --abbrev=0 --tags) --pretty=oneline --format=%ci | awk 'NR%2==0'\")\n\ndef proceedWithE2ETesting = wasTagReleasedRecently(latestTagDateString)\n\n@NonCPS\n\n def wasTagReleasedRecently(String lastTagDateString) {\n\n def formatter = DateTimeFormatter.ofPattern(\"yyyy-MM-dd HH:mm:ss Z\")\n\n def lastTagDateTime = OffsetDateTime.parse(lastTagDateString.trim(), formatter)\n\n def currentTimeMinus2hours = OffsetDateTime.now().minus(2, ChronoUnit.HOURS)\n\n return lastTagDateTime.isAfter(currentTimeMinus2hours)\n\n}If the release of core-test-service was created within the last 2 hours we will deploy it by triggering another pipeline. As the last step, it will run E2E tests. As mentioned above, both testing environments will use a shared codebase for acceptance testing. It can be done by setting up one extra job that will fetch an external repository using this block:
\ndefinition {\n cpsScm {\n scm {\n git {\n remote {\n github(\"core-e2e-tests\", 'ssh')\n credentials(\"token\")\n branch(\"master\")\n }\n extensions {\n wipeOutWorkspace()\n }\n }\n }\n\n scriptPath(\"jenkins/core_e2e_tests.gvy\")\n }\n}We chose Cucumber as our main test framework since it will give us unparalleled leverage compared to other choices. Tests are written using natural language in a way that allows for troubleshooting to be done basically at a glance. For example here is the scenario that tests the libraries responsible for sending messages in our platform exposed via HTTP endpoint:
\nScenario: Should have a working API endpoint for creating a message\n\nGiven an object with generated id and message: This is a test\nWhen making a POST call on messaging endpoint with the body containing the given object in JSON representation\nThen should result in a 201 response status code\nAnd should receive an empty bodyThe above, combined with this generated report, is a joy to work with.\n
Furthermore, we decided to create a dedicated slack channel to notify us about the new releases and test results.
\nSo after getting this message
\n![](\"https://lh5.googleusercontent.com/qyj0MbKLCEPO9M3Q8kalZoXkgfgcxddnmxk6lYUu0DrbBHhT747lCyy1SFhZDHKZXOGwUqcnsDFDPM2_rWAhnWC5l5m58RVcpsWWrBmwHIZgd3tKxSmhC6aqZHsB016Z7MXAhT0EX7mnT_N3_w\")
Sanity checks for new release train X.Y.Z on env:\n\n * REST API simple entity integration ✅\n * REST API dated entity integration ✅\n * REST API auditable entity integration\n * Messaging integration ✅\n * Messaging FIFO integration ✅\n * Storage integration ✅\n * Health integration ✅\n * Metrics integration ✅\n * Logging integration ✅We can finally get our much-needed stress-free beauty sleep, not that it will help us much, right? Hopefully, this small article will get you in the mood for some nice automation work in your company. Have fun and enjoy your peaceful nights.
","fields":{"slug":"/e2e-testing-internal-libraries/","tags":["auto1","engineering","end2end","testing","internallibraries"]}},"categoryArticles":{"edges":[{"node":{"id":"cb45e6dc-ba61-576d-adc2-ba609b0e722a","frontmatter":{"category":"Engineering","title":"Go 1.23: the new Pattern property in http.Request","date":"2024-11-26","summary":"A short review of the new Pattern property in net/http.Request","thumbnail":null,"authorName":"Andrei Gusev","authorDescription":"Senior Software Engineer at AUTO1 Group","authorAvatar":{"relativePath":"pages/go-123-the-new-pattern-property-in-http-request/avatar_Andrei.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAUBBgT/xAAWAQEBAQAAAAAAAAAAAAAAAAAAAQL/2gAMAwEAAhADEAAAAaeTsi2mDmfWV0wzf//EAB0QAAICAQUAAAAAAAAAAAAAAAECAAMEEBESEyH/2gAIAQEAAQUCtYiUvyE71ZkuQNuzaYvuQTP/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/AR//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAcEAACAgIDAAAAAAAAAAAAAAABEQACEBIhMTL/2gAIAQEABj8C4KnprGnRhqdnHWpWKPH/xAAdEAACAgMAAwAAAAAAAAAAAAABEQAhEEFRMXGB/9oACAEBAAE/IRoFjyQHGILdksEYAYLXeSuANz4mvAhI736hHP/aAAwDAQACAAMAAAAQNBc+/8QAFhEAAwAAAAAAAAAAAAAAAAAAABAR/9oACAEDAQE/ECv/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAcEAEBAAMBAAMAAAAAAAAAAAABEQAhUTEQYaH/2gAIAQEAAT8Q0H5agDhjJzCu2+nxvKetJ9/fd9x0k6GIzjdTPf69KX9xcAAKlBR2wEDDmf/Z","width":50,"height":50,"src":"/static/479e481648d69b84a342bd2575a225d4/d2d31/avatar_Andrei.jpg","srcSet":"/static/479e481648d69b84a342bd2575a225d4/d2d31/avatar_Andrei.jpg 1x,\n/static/479e481648d69b84a342bd2575a225d4/0b804/avatar_Andrei.jpg 1.5x,\n/static/479e481648d69b84a342bd2575a225d4/753c3/avatar_Andrei.jpg 2x,\n/static/479e481648d69b84a342bd2575a225d4/31ca8/avatar_Andrei.jpg 3x"}}},"headerImage":null},"html":"In Go 1.23, a new Pattern property in net/http.Request was added.\nIt contains the route pattern used to handle the request. This improvement may have gone unnoticed by many, but it can\nsimplify request processing and improve performance in cases where we need to obtain the matched routing pattern\nfor the current request. Let’s see how we can use this feature.
At AUTO1, we have more than 200 microservices, some of which are written in Go and include an HTTP server.\nIt is critically important for us to collect metrics from these services in order to quickly respond to incidents,\nmake data-driven decisions, and optimize system performance.\nFor example, we collect http_requests_total metric:
requestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{\n\t\tName: \"http_requests_total\",\n\t\tHelp: \"total request count\",\n\t}, []string{\"path\", \"method\", \"code\"})Obviously, for the path label, we should store the path template, not the specific value, for example, /test/{id} instead of /test/123.\nOtherwise, we risk a cardinality explosion, which can significantly increase the amount of stored data. You can read how to choose labels properly in \"Prometheus best practices\".
Previously, to extract the route pattern, we had to pass the multiplexer to a middleware function:
\nfunc metricsMiddleware(next http.Handler, mux *http.ServeMux) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\t_, pattern := mux.Handler(r)\n\t\t\n\t\t...\n\t\trequestsTotal.WithLabelValues(pattern, method, statusCode).Inc()\n\t}\n}In this approach, we matched the request to the route a second time, and that negatively affected performance.
\nStarting with Go 1.23, we can get the route pattern directly from the request, without passing the multiplexer to the middleware:
\nfunc middleware(f http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tpattern := r.Pattern\n\t\t\n\t\t...\n\t\trequestsTotal.WithLabelValues(pattern, method, statusCode).Inc()\n\t})\n}This significantly simplifies the middleware, making the code cleaner and more readable. We are no longer dependent on ServeMux and no longer need to re-match the request.
It is important to remember that Request.Pattern is only available after the request has been matched to a route. For example:
func main() {\n\tmux := http.NewServeMux()\n\tmux.Handle(\"/test/{id}\", routeMiddleware(handler()))\n\n\tsrv := http.Server{\n\t\tAddr: \":8080\",\n\t\tHandler: globalMiddleware(mux),\n\t}\n\n\tif err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {\n\t\tlog.Fatalf(\"server error: %s\", err)\n\t}\n}\n\nfunc routeMiddleware(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tlog.Printf(\"ROUTE pattern: %s\", r.Pattern)\n\t\th.ServeHTTP(w, r)\n\t})\n}\n\nfunc globalMiddleware(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tlog.Printf(\"GLOBAL pattern: %s\", r.Pattern)\n\t\th.ServeHTTP(w, r)\n\t\tlog.Printf(\"GLOBAL pattern after: %s\", r.Pattern)\n\t})\n}If we send a request GET http://localhost:8080/test/123, in logs we will see:
2024/11/10 13:20:37 GLOBAL pattern: \n2024/11/10 13:20:37 ROUTE pattern: /test/{id}\n2024/11/10 13:20:37 GLOBAL pattern after: /test/{id}This is the expected behavior. The multiplexer implements the http.Handler interface, and inside the ServeHTTP method, it searches for the handler that matches the request. Starting from Go 1.23, it fills the Pattern property of the request with the matching route pattern:
// net/http/server.go\nfunc (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request) {\n\t...\n\tvar h Handler\n\tif use121 {\n\t\th, _ = mux.mux121.findHandler(r)\n\t} else {\n\t\th, r.Pattern, r.pat, r.matches = mux.findHandler(r)\n\t}\n\th.ServeHTTP(w, r)\n}Thus, the r.Pattern is filled only after the ServeHTTP method of the multiplexer has been called, and the logs confirm this behavior.
In the last two Go releases, the net/http package has received significant improvements that reduced the gap with third-party packages, like gorilla/mux. Now, net/http.ServeMux supports HTTP methods and wildcards in routing patterns, bringing its functionality closer to what third-party packages previously provided.
Additionally, the introduction of the net/http.Request.Pattern property in Go 1.23 has simplified middleware development by allowing us to access the route pattern without needing to re-match the request, improving performance and reducing code complexity.
With each of these improvements, there are fewer reasons to choose third-party routers. While the choice between the standard package and third-party routers used to be more obvious, today I would recommend starting with the standard net/http package.
Paginated APIs, often encountered when dealing with servers and databases, break down large datasets into manageable chunks or pages. This methodology not only optimizes resource usage but also enhances the overall user experience by delivering content progressively. However, integrating and managing paginated data in a mobile app can be a challenge, from maintaining loading states to ensuring a seamless transition between pages and supporting content filtering.
\nIn this guide I will show you how to work with paginated APIs in Android with Jetpack Compose and Paging 3 libraries.
\n![\"Paging](\"/static/paging_demo-a325402259bf9864dced311c9eafa7c8.gif\")
\n![\"Paging](\"https://developer.android.com/static/topic/libraries/architecture/images/paging3-library-architecture.svg\")
The key components of the Paging 3 architecture include
\nWe start with the data layer and define a Retrofit API for fetching movies from network service.
\ninterface MoviesApi {\n @GET(\"/3/discover/movie?language=en-US&sort_by=popularity.desc\")\n suspend fun discover(\n @Query(\"api_key\") api_key: String,\n @Query(\"page\") page: Int,\n @Query(\"with_genres\") genres: String,\n ): Response<MoviesNetworkResponse>\n}Next, we will leverage the Room library, which is compatible with Paging 3, to create a local paging source.
\n@Dao\ninterface MoviesDao {\n @Query(\"SELECT * FROM movies ORDER BY id ASC\")\n fun getMovies(): DataSource.Factory<Int, MovieDbEntity>\n}It's important to highlight that we utilize DataSource.Factory as the return type instead of PagingSource. This choice is made specifically because DataSource.Factory offers the map() method, enabling us to map MovieDbEntity instances into domain layer models.
Now we need to put together local and remote data sources in Remote mediator.
\ngetRemoteKey().nextPage) is stored in the database after each successful network request.val page = when (loadType) {\n LoadType.REFRESH -> 1\n LoadType.PREPEND -> null\n LoadType.APPEND -> repository.getRemoteKey()?.nextPage\n } ?: return Success(endOfPaginationReached = true)For bidirectional pagination support, provide the preceding page number for the PREPEND load type. The REFRESH load type is employed when content is refreshed, initiating loading from the initial page.
\nCall our network data source method to fetch a page of data from the server.
\nval movies = moviesDataSource.discover(page)The RemoteMediator code will appear as follows:
\nclass MoviesRemoteMediator(\n private val repository: MoviesRepository,\n private val moviesDataSource: NetworkMoviesDataSource,\n) : RemoteMediator<Int, Movie>() {\n\n override suspend fun load(loadType: LoadType, state: PagingState<Int, Movie>): MediatorResult {\n\n return try {\n\n val page = when (loadType) {\n LoadType.REFRESH -> 1\n LoadType.PREPEND -> null\n LoadType.APPEND -> repository.getRemoteKey()?.nextPage\n } ?: return Success(endOfPaginationReached = true)\n\n val movies = moviesDataSource.getMovies(page)\n\n val nextPage = if (movies.isNotEmpty()) page + 1 else null\n\n repository.insertMovies(movies, nextPage, loadType == LoadType.REFRESH)\n\n Success(endOfPaginationReached = movies.isEmpty())\n\n } catch (e: Exception) {\n MediatorResult.Error(e)\n }\n }\n}RemoteMediator uses the repository to store movies list and next page number in the database.\nRepository also takes care of clearing the database table when data is refreshed:
\nclass MoviesRepositoryImpl(\n private val database: MoviesDatabase\n): MoviesRepository {\n\n override fun getMovies() = database.movies().getMovies().map { it.toDomain() }\n\n override suspend fun insertMovies(movies: List<Movie>, nextPage: Int?, clear: Boolean){\n\n database.withTransaction {\n if (clear) {\n database.movies().clear()\n }\n\n database.movies().putMovies(movies.map { it.toDbEntity() })\n\n if (nextPage != null) {\n database.remoteKey().insertKey(RemoteKey(MoviesDatabase.MOVIES_REMOTE_KEY, nextPage))\n }\n }\n }\n}The Pager object serves as a bridge between remote and local data sources. It is managing the page loading, prefetching, and handling the transitions between various data sources to provide a seamless and responsive flow of paginated content within your application. You can configure the page loading behavior with the following parameters:
\nclass MoviesScreenViewModel(\n moviesSource: MoviesRemoteMediator,\n repository: MoviesRepository,\n private val movieDetails: MovieDetailsApi,\n) : ViewModel() {\n val moviesFlow = Pager(\n config = PagingConfig(\n pageSize = PAGE_SIZE,\n prefetchDistance = PREFETCH_DISTANCE,\n initialLoadSize = INITIAL_LOAD,\n ),\n remoteMediator = moviesSource,\n pagingSourceFactory = repository.getMovies().asPagingSourceFactory()\n ).flow.cachedIn(viewModelScope)\n}Now, let's explore how to consume the paginated data in your Compose UI.
\nUse the collectAsLazyPagingItems extension function to collect the PagingData and observe changes.
val movies = viewModel.moviesFlow.collectAsLazyPagingItems()Use the LazyColumn or LazyVerticalGrid to efficiently display paginated data in your Compose UI.
LazyVerticalGrid(\n modifier = Modifier\n .fillMaxSize()\n .padding(16.dp),\n columns = GridCells.Fixed(3),\n horizontalArrangement = Arrangement.spacedBy(16.dp),\n verticalArrangement = Arrangement.spacedBy(16.dp),\n ) {\n\n items(\n count = movies.itemCount,\n key = movies.itemKey { it.id },\n ) { index ->\n\n movies[index]?.let { movie ->\n MovieCardSmall(\n movie = movie,\n onClick = { onClick(movie.id) },\n )\n }\n }\n }\n\nNote the usage of the key property for items. The key property uniquely identifies each item, allowing Compose to efficiently update and recycle composables as data changes.
\n
LazyPagingItems offers insights into the ongoing data loading process, allowing you to effectively manage errors during pagination. By examining the loadState.append property, you can determine the loading state when appending new items and present a user-friendly indication of the current status.
\nFor instance, the following code snippet demonstrates how to react to different loading states during item appending:
\nwhen (movies.loadState.append) {\n is LoadState.Loading -> { ProgressFooter() }\n is LoadState.Error -> { ErrorFooter(onRetryClick = { movies.retry() } }\n}Similarly, you can assess the loading status when the list data is refreshed by checking the loadState.refresh property. Extension functions isError() and isLoading() provide convenient checks for error and loading states, respectively.
\nfun LazyPagingItems<Movie>.isError(): Boolean = \n loadState.refresh is LoadState.Error && itemSnapshotList.isEmpty()\n\nfun LazyPagingItems<Movie>.isLoading(): Boolean = \n loadState.refresh is LoadState.Loading && itemSnapshotList.isEmpty()Incorporate these checks into your UI logic to display appropriate screens based on the loading or error states. The following code snippet demonstrates how to handle loading and error states along with providing a retry option:
\nwhen {\n movies.isLoading() -> LoadingScreen()\n movies.isError() -> ErrorScreen { movies.retry() }\n else -> MoviesGrid(movies = movies, onClick = viewModel::onMovieClick)\n}Additionally, LazyPagingItems offers a convenient retry() method, allowing you to retry the last failed data loading attempt with ease.
Now lets add filters to our movie list to see paginated list of movies filtered by genre.
\n![\"Filters](\"/static/paging_demo_filters-ff80a7606fb30961e2bda56bd6ed9985.gif\")
We start with requesting genres from the server and saving them in our database. We will keep genre data and selection in the database so that we can observe it from our domain and UI layers.
\ninterface MoviesApi {\n @GET(\"/3/genre/movie/list?language=en-US\")\n suspend fun genres(@Query(\"api_key\") api_key: String): Response<GenresNetworkResponse>\n}\n\n@Dao\ninterface GenresDao {\n\n @Insert(onConflict = OnConflictStrategy.REPLACE)\n suspend fun putGenres(genres: List<GenreDbEntity>)\n\n @Upsert(entity = GenreDbEntity::class)\n suspend fun updateGenres(genres: List<GenreDbUpdateEntity>)\n\n @Query(\"SELECT * FROM genres\")\n suspend fun getGenres(): List<GenreDbEntity>\n\n @Query(\"SELECT * FROM genres\")\n fun observeGenres(): Flow<List<GenreDbEntity>>\n}Now we can add with_genres parameter to our network request:
@GET(\"/3/discover/movie?language=en-US&sort_by=popularity.desc\")\n suspend fun discover(\n @Query(\"page\") page: Int,\n @Query(\"with_genres\") genres: String,\n @Query(\"api_key\") api_key: String,\n ): Response<MoviesNetworkResponse>Remote mediator will get selected genres from the repository
\n ...\n val genres = repository.getGenreSelection()\n val movies = moviesDataSource.getMovies(page, genres)\n ...Every time user selects a new filter we stave it in the database and restart pagination by clearing movies in the database and resetting the page to 1.
\nclass MoviesScreenViewModel {\n\n fun onGenreClick(genreItem: GenreItem) = viewModelScope.launch {\n genresProvider.setSelection(genreItem.id, !genreItem.isSelected)\n repository.clearMovies()\n }\n \n}class MoviesRepositoryImpl {\n\n override suspend fun clearMovies() = withContext(Dispatchers.IO) {\n database.withTransaction {\n database.movies().clear()\n database.remoteKey().insertKey(RemoteKey(MoviesDatabase.MOVIES_REMOTE_KEY, 1))\n }\n }\n\n}
Source code: https://github.com/auto1-oss/android-compose-paging\n
Official documentation: https://developer.android.com/topic/libraries/architecture/paging/v3-overview
Sense of Security
\nTreat the injured first
\nTrust your tools
\nCommunication via Radio
\nWhat I present today is how to use police techniques and their mindset to detect and solve bugs in software, so that you can be the best software developer that society can provide.
\nFor some years, I worked as a police officer in Brazil and I can say that it is a challenging career. The first step was to pass a national test, where knowledge of various laws was required. Criminal laws, traffic laws, and even the laws of physics.
\nThe second step was to pass a battery of medical exams. Almost perfect health was necessary to be able to practice the profession, in addition to having most of the parts of the human body.
\nThe third step was the psychological examination. It was necessary to have a clear mind and not have impulses different from those expected from a person who must protect other people.
\nAs a fourth part, there was also the fitness test, a test in which career candidates had to prove they had sufficient strength and speed to carry out daily police tasks. Running, pull-ups, sit-ups and long jump are examples of what was required.
\nThe last stage was the Police Academy, where for three months those from now on known as students would learn and prove the learning of several new skills: first aid, use of firearms (pistols, carbines, submachine guns, etc.), non-lethal weapons, martial arts, public policy, and many other disciplines.
\nAll this so that society receives the best police officer that its people can provide. So, what about using some of the following principles and techniques to become the best software developer that society can provide?
\nWithout needing to be a security expert, you already know that it is impossible to have a completely safe city. Even if we reached the point of having perfect police officers, which in itself is impossible, we would not have a police officer for every citizen. In general, the recommendation is 300 police officers for every 100 thousand inhabitants.
\nAware of this limitation, that it is not possible to guarantee complete security for all citizens, police agencies work with the concept of a sense of security. Through strategic actions to combat crime, citizens' feeling that they are safe is increased. This mental condition gives them peace of mind when working, studying, and carrying out their daily tasks.
\nThis sense of security also affects criminals. The common visibility of police officers carrying out their duties in strategic places, as well as effective actions to combat crime, make criminals stay away from that place.
\nNotice that ending 100% of crime is impossible, what we do well is limit its actions. From here we can extract our first lesson when dealing with software: perfect code, totally bug-free, is impossible.
\nDespite this conclusion, the fact that it is impossible does not mean that we should simply ignore the problem. On the contrary, we can, like the police, create strategic actions to combat Bugs. To the point that Bugs themselves will run away from our code.
\nTest Driven Development - TDD is a strategic action that provides a feeling of security in your code. Before the crime occurs, you position police officers in a specific region. Before the Bug occurs, you position the test in a specific place in your code.
\nRobert C. Martin, in his book \"Clean Code: A HandBook of Agile Software Craftsmanship\", teaches us that writing unit tests before writing production code is just the tip of the iceberg in defining TDD. There must be a continuous cycle of improvement, in which there is always a test covering the code you are developing and this security that the code is protected allows you to play with it, and work with it. After all, whatever problem may happen, there is a police-test there to protect you.
\nOne of the functions of Highway Police Officers is to provide first aid in the event of traffic accidents. There are often serious injuries that must be treated as urgently as possible, as even 1 minute can make the difference between life and death.
\nThe first step when arriving at a traffic accident site is to mark the location appropriately to prevent other accidents. We must use lighting devices and other appliances to make it clear to other vehicles that there is a danger there.
\nThe police officer must then check the condition of all the injured to prioritize them according to the severity of each situation.
\nOnce the treatment of each injured person has been completed and, if necessary, they have been taken to the hospital, it is time to survey the accident site. At this stage, the police officer begins to collect all the information relating to the accident to have sufficient information so that a decision can be made on the cause of the accident and, when applicable, any culprits.
\nThis is the mental model that you and I must adopt when we are faced with a Bug. Although several procedures have been adopted to prevent it from occurring, it is a fatality that can happen. It's like when the highway was well designed, the vehicles were up to date with maintenance, but even so, due to carelessness or misunderstanding on the part of the driver, an accident happens.
\nAs in the case of traffic accidents, the first step to take is to secure the environment so that the Bug does not cause other problems. If possible, we can revert the application version to one in which the Bug did not appear. We can use a feature flag to temporarily disable a feature. Regardless of the method used to stop the bleeding, the important thing is that we must focus on treating the injured first rather than looking for culprits.
\nWe have to agree with David Thomas and Andrew Hunt, in their work \"The Pragmatic Programmer: your journey to mastery\" that facing Bugs is a sensitive subject for most developers. Instead of treating the issue as a problem to be solved, some approach it with denial, pointing fingers, lame excuses, or simply apathy. It's exactly the opposite of what is expected!
\nInstead of looking for the culprit in a traffic accident, we must first assist the victims and stop the bleeding. In software, instead of finding out who inserted the Bug, we must solve it.
\nWhen treating an injured person in an emergency, one approach is to use the Glasgow Scale to determine the patient's consciousness. On this scale, one of the components is the verbal response.
\nThe injured person is asked simple questions, and by analyzing the responses, we can assess the level of consciousness and consequently the level of urgency of their care.
\nYou and I must carefully read every error message that is displayed during the Bug, it may contain vital information so that we can resolve the problem.
\nYou may be thinking, but what if there is no error message? If we are just faced with an incorrect result? Well, this also happens in traffic accidents, it is not uncommon for there to be an unconscious victim who cannot answer questions.
\nWhen we discussed the sense of security, we talked about the importance of tests in our application. Go and build a test that can reproduce that Bug, so you can extract the information you need to solve it.
\nThink of these tests like the motor tests that highway patrol officers perform on unconscious victims to check their level of consciousness: do the eyes open in response to pain? Is there a pupillary response to light stimulation? Mentally translate this into software language: Does the Bug repeat itself when I enter this and that parameter? And if I insert this new argument, does it happen too?
\nTo decide on a Bug, data is needed. If you don't receive this data, you must extract it.
\nPolice work is very complex and your day-to-day job involves the fight for life and protection of the most important assets for society. There would be no point in having a modern and efficient selection and training process for the development of police officers if they were not given the appropriate tools for their work.
\nAnd when I say appropriate, I don't mean they are minimally capable of performing the task, as the result delivered is directly related to the quality and safety of the tools. For an excellent result, not only the police officers but their tools must also be excellent.
\nIn the police agency I worked for, one of the work tools used was an Austrian pistol known worldwide for its safety and efficiency. The Glock 17 pistol was developed to meet the strict criteria of the Austrian army who wanted to replace the model used during the Second World War.
\nIn addition to requirements on durability, efficiency, and reliability, safety against accidental firing was one of the pivots of the new model.
\nBut how did a knife factory manage to deliver the pistol model that would prove to be the safest? Well, it can be said that, in addition to intense testing, the secret to reliability was simplicity and maintainability. The person responsible for the project designed the equipment with as few parts as possible, minimizing complexity. Today the Glock pistol is made with an average of 35 parts, which is a much smaller quantity than other pistols on the market.
\nWith a simpler design, so-called first-level maintenance is much easier. Equipment users can easily perform maintenance procedures, making the equipment's efficiency last for decades.
\nOf course, you must already remember the principle with the acronym KISS, which stands for Keep it Simple (omitting the last word because I don't want to offend the reader). The secret of that renowned pistol model was not in the used materials, but in being simple. The secret of maintaining your software in good shape is to keep it simple, having a specific component for each task. When a problem arises, you will know exactly which component to look for, as each component has a responsibility, this is the principle of Single Responsibility.
\nBecause of all this, when you are at a shooting range practicing your aim, the police officer knows that if he is not hitting the target, most likely, the error is in his procedure and not in the equipment.
\nAllan Kelly, in one of his contributions to the Book \"97 Things Every Programmer Should Know\", corroborates this idea that we should trust our tools when dealing with Bugs. Considering our tools are widely used, mature, and in use by multiple companies, we have little reason to doubt their quality.
\nOf course, a prototype, version 0.1, is much more fragile than a library that has been on the market for several years. We are also not saying that it is impossible to have bugs in compilers.
\nBut considering how rare compiler bugs are, what we are saying is that in the search for a Bug solution, we should trust our tools and look at our piece of code first.
\nImagine it as an efficient distribution of efforts. The library or compiler code has already been reviewed and tested for many years by countless people. As for that little piece of code of yours, only God knows who besides you has looked at it on your team.
\nOn one of those days, I managed to solve an unpleasant bug that occurred in one of our services. It was a non-deterministic error in our test suite, that is, the Bug only pushed out its head when no one was looking.
\nThis type of non-deterministic error generally involves three types of origin: multithread, time sensitivity, and dissynchrony between construction and cleaning of test data.
\nAnalyzing the body of the failed test, after several hours, of course, I began to strongly distrust the way we created timestamps with a native Java library. After all, the test seemed perfect to me, if there was an error it must have been somewhere else.
\nBut, aware of this lesson that we should trust our tools, especially since the component in question had been in Java for 9 years, I ignored this distrust. This saved me a lot of time and embarrassment, as I'm going to share that the error was elsewhere.
\nOur test suite ran all tests in a single thread, so it wouldn't be a concurrency issue. But it also ran each test in a deterministic but unpredictable order. Therefore, the source of the intermittent problem must be in some specific order in which the tests were performed.
\nAfter several hours of analyzing the logs, I was able to identify two tests that failed when run in sequence. It turned out that, although our tests performed a test data cleaning routine after each execution, a simple typo had silently overwritten the cleaning method in one of the tests, which generated data inconsistency if such tests were executed in a specific order.
\nI remember that one of the first systems I developed when I worked in the police was one to replace the sending of statistics through radio communication. Many years ago, each police station was supposed to send daily work statistics via radio. Data such as number of people and vehicles inspected, car accidents, etc.
\nThis took a lot of time as the audio quality of radio communications was not always ideal. The interlocutors spent a lot of time transmitting and confirming each piece of information. All this using Q-Code
\nThe Q-Code is a standardized collection of three-letter codes used in radio communication. The use of Q-code even in voice transmissions makes communication safer and more efficient. Each set of three letters, always starting with the letter Q, has a specific meaning that is known to the interlocutors.
\nI could see that two of those codes were widely used among police officers in radio communication: QAP and QSL.
\nQAP was used when you wanted to transmit a message. Let's say someone wanted to talk to me, so he said on the radio: \"Bruno, QAP?\". This could be translated as \"Bruno, are you ready to hear a message?\"
\nMy answer could be QAP or for example QRM. If I responded with QAP, I was signaling that yes, I was ready to receive the message. If you responded with QRM, I would be informing you that at that moment the communication is being interfered with.
\nDuring communication, the second most used Q-Code appears, QSL. At the end of each message, the sender ends it with the QSL. This means that he is asking the receiver to confirm whether they received the message. It would be something like: \"There is a suspicious vehicle heading towards your team, QSL?\" If I had understood the entire message, I should respond with a simple \"QSL\". In the absence of a response, the sender repeats the message until receiving confirmation from the receiver.
\nEach message is concluded with a QSL, and its reception is confirmed with a QSL. Does this remind you of any transmission control protocol out here, used on the internet?
\nThe main thing here is that a large amount of knowledge is condensed into three letters, in a language that provides interlocutors with efficiency and security. Eric Evans, the author of \"Domain-Driven Design: Tackling Complexity in the Heart of Software\", would agree with me that there is a principle here that should also be adopted in software development.
\nA language specific to the domain, which considers possible interference from the environment, a ubiquitous language that is known by all interlocutors, can be the difference between life and death, between success and damnation of a project.
\nWhen a project does not have its own language for that domain, this creates the need for numerous translations between interlocutors. Developers have to translate for domain experts. These, in turn, have to translate their demands to developers. Even among developers, there may be a need for translations.
\nAll these translations make concepts confusing and the consequences of a lack of mutual understanding can be disastrous.
\nThe importance of this in trying to arrest a Bug goes much further than compilation errors or exceptions. It goes through the swampy characteristic that the software may be logically working almost perfectly, but it is not delivering what was expected by the stakeholder.
\nLet's look at the following pieces of code:
\nif (order.status != completed) {\n // (...)\n}if (order.isFullyRefunded()) {\n // (...)\n}Here we have a small example of two ways of expressing the same situation. In the second piece of code, a common language between developers and domain experts was used. Here, possible logical errors can be easily noticed, without the need to keep in mind what the possible states of the Order are and what the expected state is for that case.
\nSee the following code:
\nPayment newPayment = payment.add(10);What do you expect from this code? Does it add ten dollars to the payment? Ten transactions, inbound or outbound? Will the method create a new payment object or will it just change the value of the payment variable and create a new reference called newPayment?
\nA lesson also taken from the book \"Clean Code: A HandBook of Agile Software Craftsmanship\" shows that clear communication must also take place between software developers. Functions must communicate exactly what they are doing.
\nIf you have to look at the function's implementation to know what it's actually doing, that's a big sign that you need to rename it.
\nI'm very grateful that you've made it this far, you've certainly learned a lot about police activity and how some of its principles can be applied to combat and remove Bugs from your software.
\nBy applying this knowledge you will be able to work more confidently in your code, giving you peace in your daily life and also allowing you to experiment and discover new things.
\nYour aim will be improved and you will get straight to the point, avoiding damage instead of first looking for the culprits. You become bolder and go further because you know you can trust your tools. This leaves you more time to refine your code and seek excellence.
\nFinally, not only will your code be safe and efficient, but its result will also be in line with what your customers expect because through effective communication you can accurately deliver what was demanded from you.
\nThe result of all this is that Bug is in jail, lives are no longer in danger, and you can go home at the end of your shift to enjoy the rest of the day with your family which is looking forward to welcoming their hero back.
","fields":{"slug":"/bug-police-officer/","tags":["debugging","testing","TDD","DDD"]}}}]},"authorArticles":null},"pageContext":{"slug":"/e2e-testing-internal-libraries/","tags":["auto1","engineering","end2end","testing","internallibraries"],"category":"Engineering","author":"Mikołaj Kłosowski","date":"2022-07-18"}}