![\"Paging](\"/static/paging_demo-a325402259bf9864dced311c9eafa7c8.gif\")
In Go 1.23, a new Pattern property in net/http.Request was added.\nIt contains the route pattern used to handle the request. This improvement may have gone unnoticed by many, but it can\nsimplify request processing and improve performance in cases where we need to obtain the matched routing pattern\nfor the current request. Let’s see how we can use this feature.
At AUTO1, we have more than 200 microservices, some of which are written in Go and include an HTTP server.\nIt is critically important for us to collect metrics from these services in order to quickly respond to incidents,\nmake data-driven decisions, and optimize system performance.\nFor example, we collect http_requests_total metric:
requestsTotal = promauto.NewCounterVec(prometheus.CounterOpts{\n\t\tName: \"http_requests_total\",\n\t\tHelp: \"total request count\",\n\t}, []string{\"path\", \"method\", \"code\"})Obviously, for the path label, we should store the path template, not the specific value, for example, /test/{id} instead of /test/123.\nOtherwise, we risk a cardinality explosion, which can significantly increase the amount of stored data. You can read how to choose labels properly in \"Prometheus best practices\".
Previously, to extract the route pattern, we had to pass the multiplexer to a middleware function:
\nfunc metricsMiddleware(next http.Handler, mux *http.ServeMux) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\t_, pattern := mux.Handler(r)\n\t\t\n\t\t...\n\t\trequestsTotal.WithLabelValues(pattern, method, statusCode).Inc()\n\t}\n}In this approach, we matched the request to the route a second time, and that negatively affected performance.
\nStarting with Go 1.23, we can get the route pattern directly from the request, without passing the multiplexer to the middleware:
\nfunc middleware(f http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tpattern := r.Pattern\n\t\t\n\t\t...\n\t\trequestsTotal.WithLabelValues(pattern, method, statusCode).Inc()\n\t})\n}This significantly simplifies the middleware, making the code cleaner and more readable. We are no longer dependent on ServeMux and no longer need to re-match the request.
It is important to remember that Request.Pattern is only available after the request has been matched to a route. For example:
func main() {\n\tmux := http.NewServeMux()\n\tmux.Handle(\"/test/{id}\", routeMiddleware(handler()))\n\n\tsrv := http.Server{\n\t\tAddr: \":8080\",\n\t\tHandler: globalMiddleware(mux),\n\t}\n\n\tif err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {\n\t\tlog.Fatalf(\"server error: %s\", err)\n\t}\n}\n\nfunc routeMiddleware(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tlog.Printf(\"ROUTE pattern: %s\", r.Pattern)\n\t\th.ServeHTTP(w, r)\n\t})\n}\n\nfunc globalMiddleware(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tlog.Printf(\"GLOBAL pattern: %s\", r.Pattern)\n\t\th.ServeHTTP(w, r)\n\t\tlog.Printf(\"GLOBAL pattern after: %s\", r.Pattern)\n\t})\n}If we send a request GET http://localhost:8080/test/123, in logs we will see:
2024/11/10 13:20:37 GLOBAL pattern: \n2024/11/10 13:20:37 ROUTE pattern: /test/{id}\n2024/11/10 13:20:37 GLOBAL pattern after: /test/{id}This is the expected behavior. The multiplexer implements the http.Handler interface, and inside the ServeHTTP method, it searches for the handler that matches the request. Starting from Go 1.23, it fills the Pattern property of the request with the matching route pattern:
// net/http/server.go\nfunc (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request) {\n\t...\n\tvar h Handler\n\tif use121 {\n\t\th, _ = mux.mux121.findHandler(r)\n\t} else {\n\t\th, r.Pattern, r.pat, r.matches = mux.findHandler(r)\n\t}\n\th.ServeHTTP(w, r)\n}Thus, the r.Pattern is filled only after the ServeHTTP method of the multiplexer has been called, and the logs confirm this behavior.
In the last two Go releases, the net/http package has received significant improvements that reduced the gap with third-party packages, like gorilla/mux. Now, net/http.ServeMux supports HTTP methods and wildcards in routing patterns, bringing its functionality closer to what third-party packages previously provided.
Additionally, the introduction of the net/http.Request.Pattern property in Go 1.23 has simplified middleware development by allowing us to access the route pattern without needing to re-match the request, improving performance and reducing code complexity.
With each of these improvements, there are fewer reasons to choose third-party routers. While the choice between the standard package and third-party routers used to be more obvious, today I would recommend starting with the standard net/http package.
Paginated APIs, often encountered when dealing with servers and databases, break down large datasets into manageable chunks or pages. This methodology not only optimizes resource usage but also enhances the overall user experience by delivering content progressively. However, integrating and managing paginated data in a mobile app can be a challenge, from maintaining loading states to ensuring a seamless transition between pages and supporting content filtering.
\nIn this guide I will show you how to work with paginated APIs in Android with Jetpack Compose and Paging 3 libraries.
\n
\n![\"Paging](\"https://developer.android.com/static/topic/libraries/architecture/images/paging3-library-architecture.svg\")
The key components of the Paging 3 architecture include
\nWe start with the data layer and define a Retrofit API for fetching movies from network service.
\ninterface MoviesApi {\n @GET(\"/3/discover/movie?language=en-US&sort_by=popularity.desc\")\n suspend fun discover(\n @Query(\"api_key\") api_key: String,\n @Query(\"page\") page: Int,\n @Query(\"with_genres\") genres: String,\n ): Response<MoviesNetworkResponse>\n}Next, we will leverage the Room library, which is compatible with Paging 3, to create a local paging source.
\n@Dao\ninterface MoviesDao {\n @Query(\"SELECT * FROM movies ORDER BY id ASC\")\n fun getMovies(): DataSource.Factory<Int, MovieDbEntity>\n}It's important to highlight that we utilize DataSource.Factory as the return type instead of PagingSource. This choice is made specifically because DataSource.Factory offers the map() method, enabling us to map MovieDbEntity instances into domain layer models.
Now we need to put together local and remote data sources in Remote mediator.
\ngetRemoteKey().nextPage) is stored in the database after each successful network request.val page = when (loadType) {\n LoadType.REFRESH -> 1\n LoadType.PREPEND -> null\n LoadType.APPEND -> repository.getRemoteKey()?.nextPage\n } ?: return Success(endOfPaginationReached = true)For bidirectional pagination support, provide the preceding page number for the PREPEND load type. The REFRESH load type is employed when content is refreshed, initiating loading from the initial page.
\nCall our network data source method to fetch a page of data from the server.
\nval movies = moviesDataSource.discover(page)The RemoteMediator code will appear as follows:
\nclass MoviesRemoteMediator(\n private val repository: MoviesRepository,\n private val moviesDataSource: NetworkMoviesDataSource,\n) : RemoteMediator<Int, Movie>() {\n\n override suspend fun load(loadType: LoadType, state: PagingState<Int, Movie>): MediatorResult {\n\n return try {\n\n val page = when (loadType) {\n LoadType.REFRESH -> 1\n LoadType.PREPEND -> null\n LoadType.APPEND -> repository.getRemoteKey()?.nextPage\n } ?: return Success(endOfPaginationReached = true)\n\n val movies = moviesDataSource.getMovies(page)\n\n val nextPage = if (movies.isNotEmpty()) page + 1 else null\n\n repository.insertMovies(movies, nextPage, loadType == LoadType.REFRESH)\n\n Success(endOfPaginationReached = movies.isEmpty())\n\n } catch (e: Exception) {\n MediatorResult.Error(e)\n }\n }\n}RemoteMediator uses the repository to store movies list and next page number in the database.\nRepository also takes care of clearing the database table when data is refreshed:
\nclass MoviesRepositoryImpl(\n private val database: MoviesDatabase\n): MoviesRepository {\n\n override fun getMovies() = database.movies().getMovies().map { it.toDomain() }\n\n override suspend fun insertMovies(movies: List<Movie>, nextPage: Int?, clear: Boolean){\n\n database.withTransaction {\n if (clear) {\n database.movies().clear()\n }\n\n database.movies().putMovies(movies.map { it.toDbEntity() })\n\n if (nextPage != null) {\n database.remoteKey().insertKey(RemoteKey(MoviesDatabase.MOVIES_REMOTE_KEY, nextPage))\n }\n }\n }\n}The Pager object serves as a bridge between remote and local data sources. It is managing the page loading, prefetching, and handling the transitions between various data sources to provide a seamless and responsive flow of paginated content within your application. You can configure the page loading behavior with the following parameters:
\nclass MoviesScreenViewModel(\n moviesSource: MoviesRemoteMediator,\n repository: MoviesRepository,\n private val movieDetails: MovieDetailsApi,\n) : ViewModel() {\n val moviesFlow = Pager(\n config = PagingConfig(\n pageSize = PAGE_SIZE,\n prefetchDistance = PREFETCH_DISTANCE,\n initialLoadSize = INITIAL_LOAD,\n ),\n remoteMediator = moviesSource,\n pagingSourceFactory = repository.getMovies().asPagingSourceFactory()\n ).flow.cachedIn(viewModelScope)\n}Now, let's explore how to consume the paginated data in your Compose UI.
\nUse the collectAsLazyPagingItems extension function to collect the PagingData and observe changes.
val movies = viewModel.moviesFlow.collectAsLazyPagingItems()Use the LazyColumn or LazyVerticalGrid to efficiently display paginated data in your Compose UI.
LazyVerticalGrid(\n modifier = Modifier\n .fillMaxSize()\n .padding(16.dp),\n columns = GridCells.Fixed(3),\n horizontalArrangement = Arrangement.spacedBy(16.dp),\n verticalArrangement = Arrangement.spacedBy(16.dp),\n ) {\n\n items(\n count = movies.itemCount,\n key = movies.itemKey { it.id },\n ) { index ->\n\n movies[index]?.let { movie ->\n MovieCardSmall(\n movie = movie,\n onClick = { onClick(movie.id) },\n )\n }\n }\n }\n\nNote the usage of the key property for items. The key property uniquely identifies each item, allowing Compose to efficiently update and recycle composables as data changes.
\n
LazyPagingItems offers insights into the ongoing data loading process, allowing you to effectively manage errors during pagination. By examining the loadState.append property, you can determine the loading state when appending new items and present a user-friendly indication of the current status.
\nFor instance, the following code snippet demonstrates how to react to different loading states during item appending:
\nwhen (movies.loadState.append) {\n is LoadState.Loading -> { ProgressFooter() }\n is LoadState.Error -> { ErrorFooter(onRetryClick = { movies.retry() } }\n}Similarly, you can assess the loading status when the list data is refreshed by checking the loadState.refresh property. Extension functions isError() and isLoading() provide convenient checks for error and loading states, respectively.
\nfun LazyPagingItems<Movie>.isError(): Boolean = \n loadState.refresh is LoadState.Error && itemSnapshotList.isEmpty()\n\nfun LazyPagingItems<Movie>.isLoading(): Boolean = \n loadState.refresh is LoadState.Loading && itemSnapshotList.isEmpty()Incorporate these checks into your UI logic to display appropriate screens based on the loading or error states. The following code snippet demonstrates how to handle loading and error states along with providing a retry option:
\nwhen {\n movies.isLoading() -> LoadingScreen()\n movies.isError() -> ErrorScreen { movies.retry() }\n else -> MoviesGrid(movies = movies, onClick = viewModel::onMovieClick)\n}Additionally, LazyPagingItems offers a convenient retry() method, allowing you to retry the last failed data loading attempt with ease.
Now lets add filters to our movie list to see paginated list of movies filtered by genre.
\n![\"Filters](\"/static/paging_demo_filters-ff80a7606fb30961e2bda56bd6ed9985.gif\")
We start with requesting genres from the server and saving them in our database. We will keep genre data and selection in the database so that we can observe it from our domain and UI layers.
\ninterface MoviesApi {\n @GET(\"/3/genre/movie/list?language=en-US\")\n suspend fun genres(@Query(\"api_key\") api_key: String): Response<GenresNetworkResponse>\n}\n\n@Dao\ninterface GenresDao {\n\n @Insert(onConflict = OnConflictStrategy.REPLACE)\n suspend fun putGenres(genres: List<GenreDbEntity>)\n\n @Upsert(entity = GenreDbEntity::class)\n suspend fun updateGenres(genres: List<GenreDbUpdateEntity>)\n\n @Query(\"SELECT * FROM genres\")\n suspend fun getGenres(): List<GenreDbEntity>\n\n @Query(\"SELECT * FROM genres\")\n fun observeGenres(): Flow<List<GenreDbEntity>>\n}Now we can add with_genres parameter to our network request:
@GET(\"/3/discover/movie?language=en-US&sort_by=popularity.desc\")\n suspend fun discover(\n @Query(\"page\") page: Int,\n @Query(\"with_genres\") genres: String,\n @Query(\"api_key\") api_key: String,\n ): Response<MoviesNetworkResponse>Remote mediator will get selected genres from the repository
\n ...\n val genres = repository.getGenreSelection()\n val movies = moviesDataSource.getMovies(page, genres)\n ...Every time user selects a new filter we stave it in the database and restart pagination by clearing movies in the database and resetting the page to 1.
\nclass MoviesScreenViewModel {\n\n fun onGenreClick(genreItem: GenreItem) = viewModelScope.launch {\n genresProvider.setSelection(genreItem.id, !genreItem.isSelected)\n repository.clearMovies()\n }\n \n}class MoviesRepositoryImpl {\n\n override suspend fun clearMovies() = withContext(Dispatchers.IO) {\n database.withTransaction {\n database.movies().clear()\n database.remoteKey().insertKey(RemoteKey(MoviesDatabase.MOVIES_REMOTE_KEY, 1))\n }\n }\n\n}
Source code: https://github.com/auto1-oss/android-compose-paging\n
Official documentation: https://developer.android.com/topic/libraries/architecture/paging/v3-overview
Sense of Security
\nTreat the injured first
\nTrust your tools
\nCommunication via Radio
\nWhat I present today is how to use police techniques and their mindset to detect and solve bugs in software, so that you can be the best software developer that society can provide.
\nFor some years, I worked as a police officer in Brazil and I can say that it is a challenging career. The first step was to pass a national test, where knowledge of various laws was required. Criminal laws, traffic laws, and even the laws of physics.
\nThe second step was to pass a battery of medical exams. Almost perfect health was necessary to be able to practice the profession, in addition to having most of the parts of the human body.
\nThe third step was the psychological examination. It was necessary to have a clear mind and not have impulses different from those expected from a person who must protect other people.
\nAs a fourth part, there was also the fitness test, a test in which career candidates had to prove they had sufficient strength and speed to carry out daily police tasks. Running, pull-ups, sit-ups and long jump are examples of what was required.
\nThe last stage was the Police Academy, where for three months those from now on known as students would learn and prove the learning of several new skills: first aid, use of firearms (pistols, carbines, submachine guns, etc.), non-lethal weapons, martial arts, public policy, and many other disciplines.
\nAll this so that society receives the best police officer that its people can provide. So, what about using some of the following principles and techniques to become the best software developer that society can provide?
\nWithout needing to be a security expert, you already know that it is impossible to have a completely safe city. Even if we reached the point of having perfect police officers, which in itself is impossible, we would not have a police officer for every citizen. In general, the recommendation is 300 police officers for every 100 thousand inhabitants.
\nAware of this limitation, that it is not possible to guarantee complete security for all citizens, police agencies work with the concept of a sense of security. Through strategic actions to combat crime, citizens' feeling that they are safe is increased. This mental condition gives them peace of mind when working, studying, and carrying out their daily tasks.
\nThis sense of security also affects criminals. The common visibility of police officers carrying out their duties in strategic places, as well as effective actions to combat crime, make criminals stay away from that place.
\nNotice that ending 100% of crime is impossible, what we do well is limit its actions. From here we can extract our first lesson when dealing with software: perfect code, totally bug-free, is impossible.
\nDespite this conclusion, the fact that it is impossible does not mean that we should simply ignore the problem. On the contrary, we can, like the police, create strategic actions to combat Bugs. To the point that Bugs themselves will run away from our code.
\nTest Driven Development - TDD is a strategic action that provides a feeling of security in your code. Before the crime occurs, you position police officers in a specific region. Before the Bug occurs, you position the test in a specific place in your code.
\nRobert C. Martin, in his book \"Clean Code: A HandBook of Agile Software Craftsmanship\", teaches us that writing unit tests before writing production code is just the tip of the iceberg in defining TDD. There must be a continuous cycle of improvement, in which there is always a test covering the code you are developing and this security that the code is protected allows you to play with it, and work with it. After all, whatever problem may happen, there is a police-test there to protect you.
\nOne of the functions of Highway Police Officers is to provide first aid in the event of traffic accidents. There are often serious injuries that must be treated as urgently as possible, as even 1 minute can make the difference between life and death.
\nThe first step when arriving at a traffic accident site is to mark the location appropriately to prevent other accidents. We must use lighting devices and other appliances to make it clear to other vehicles that there is a danger there.
\nThe police officer must then check the condition of all the injured to prioritize them according to the severity of each situation.
\nOnce the treatment of each injured person has been completed and, if necessary, they have been taken to the hospital, it is time to survey the accident site. At this stage, the police officer begins to collect all the information relating to the accident to have sufficient information so that a decision can be made on the cause of the accident and, when applicable, any culprits.
\nThis is the mental model that you and I must adopt when we are faced with a Bug. Although several procedures have been adopted to prevent it from occurring, it is a fatality that can happen. It's like when the highway was well designed, the vehicles were up to date with maintenance, but even so, due to carelessness or misunderstanding on the part of the driver, an accident happens.
\nAs in the case of traffic accidents, the first step to take is to secure the environment so that the Bug does not cause other problems. If possible, we can revert the application version to one in which the Bug did not appear. We can use a feature flag to temporarily disable a feature. Regardless of the method used to stop the bleeding, the important thing is that we must focus on treating the injured first rather than looking for culprits.
\nWe have to agree with David Thomas and Andrew Hunt, in their work \"The Pragmatic Programmer: your journey to mastery\" that facing Bugs is a sensitive subject for most developers. Instead of treating the issue as a problem to be solved, some approach it with denial, pointing fingers, lame excuses, or simply apathy. It's exactly the opposite of what is expected!
\nInstead of looking for the culprit in a traffic accident, we must first assist the victims and stop the bleeding. In software, instead of finding out who inserted the Bug, we must solve it.
\nWhen treating an injured person in an emergency, one approach is to use the Glasgow Scale to determine the patient's consciousness. On this scale, one of the components is the verbal response.
\nThe injured person is asked simple questions, and by analyzing the responses, we can assess the level of consciousness and consequently the level of urgency of their care.
\nYou and I must carefully read every error message that is displayed during the Bug, it may contain vital information so that we can resolve the problem.
\nYou may be thinking, but what if there is no error message? If we are just faced with an incorrect result? Well, this also happens in traffic accidents, it is not uncommon for there to be an unconscious victim who cannot answer questions.
\nWhen we discussed the sense of security, we talked about the importance of tests in our application. Go and build a test that can reproduce that Bug, so you can extract the information you need to solve it.
\nThink of these tests like the motor tests that highway patrol officers perform on unconscious victims to check their level of consciousness: do the eyes open in response to pain? Is there a pupillary response to light stimulation? Mentally translate this into software language: Does the Bug repeat itself when I enter this and that parameter? And if I insert this new argument, does it happen too?
\nTo decide on a Bug, data is needed. If you don't receive this data, you must extract it.
\nPolice work is very complex and your day-to-day job involves the fight for life and protection of the most important assets for society. There would be no point in having a modern and efficient selection and training process for the development of police officers if they were not given the appropriate tools for their work.
\nAnd when I say appropriate, I don't mean they are minimally capable of performing the task, as the result delivered is directly related to the quality and safety of the tools. For an excellent result, not only the police officers but their tools must also be excellent.
\nIn the police agency I worked for, one of the work tools used was an Austrian pistol known worldwide for its safety and efficiency. The Glock 17 pistol was developed to meet the strict criteria of the Austrian army who wanted to replace the model used during the Second World War.
\nIn addition to requirements on durability, efficiency, and reliability, safety against accidental firing was one of the pivots of the new model.
\nBut how did a knife factory manage to deliver the pistol model that would prove to be the safest? Well, it can be said that, in addition to intense testing, the secret to reliability was simplicity and maintainability. The person responsible for the project designed the equipment with as few parts as possible, minimizing complexity. Today the Glock pistol is made with an average of 35 parts, which is a much smaller quantity than other pistols on the market.
\nWith a simpler design, so-called first-level maintenance is much easier. Equipment users can easily perform maintenance procedures, making the equipment's efficiency last for decades.
\nOf course, you must already remember the principle with the acronym KISS, which stands for Keep it Simple (omitting the last word because I don't want to offend the reader). The secret of that renowned pistol model was not in the used materials, but in being simple. The secret of maintaining your software in good shape is to keep it simple, having a specific component for each task. When a problem arises, you will know exactly which component to look for, as each component has a responsibility, this is the principle of Single Responsibility.
\nBecause of all this, when you are at a shooting range practicing your aim, the police officer knows that if he is not hitting the target, most likely, the error is in his procedure and not in the equipment.
\nAllan Kelly, in one of his contributions to the Book \"97 Things Every Programmer Should Know\", corroborates this idea that we should trust our tools when dealing with Bugs. Considering our tools are widely used, mature, and in use by multiple companies, we have little reason to doubt their quality.
\nOf course, a prototype, version 0.1, is much more fragile than a library that has been on the market for several years. We are also not saying that it is impossible to have bugs in compilers.
\nBut considering how rare compiler bugs are, what we are saying is that in the search for a Bug solution, we should trust our tools and look at our piece of code first.
\nImagine it as an efficient distribution of efforts. The library or compiler code has already been reviewed and tested for many years by countless people. As for that little piece of code of yours, only God knows who besides you has looked at it on your team.
\nOn one of those days, I managed to solve an unpleasant bug that occurred in one of our services. It was a non-deterministic error in our test suite, that is, the Bug only pushed out its head when no one was looking.
\nThis type of non-deterministic error generally involves three types of origin: multithread, time sensitivity, and dissynchrony between construction and cleaning of test data.
\nAnalyzing the body of the failed test, after several hours, of course, I began to strongly distrust the way we created timestamps with a native Java library. After all, the test seemed perfect to me, if there was an error it must have been somewhere else.
\nBut, aware of this lesson that we should trust our tools, especially since the component in question had been in Java for 9 years, I ignored this distrust. This saved me a lot of time and embarrassment, as I'm going to share that the error was elsewhere.
\nOur test suite ran all tests in a single thread, so it wouldn't be a concurrency issue. But it also ran each test in a deterministic but unpredictable order. Therefore, the source of the intermittent problem must be in some specific order in which the tests were performed.
\nAfter several hours of analyzing the logs, I was able to identify two tests that failed when run in sequence. It turned out that, although our tests performed a test data cleaning routine after each execution, a simple typo had silently overwritten the cleaning method in one of the tests, which generated data inconsistency if such tests were executed in a specific order.
\nI remember that one of the first systems I developed when I worked in the police was one to replace the sending of statistics through radio communication. Many years ago, each police station was supposed to send daily work statistics via radio. Data such as number of people and vehicles inspected, car accidents, etc.
\nThis took a lot of time as the audio quality of radio communications was not always ideal. The interlocutors spent a lot of time transmitting and confirming each piece of information. All this using Q-Code
\nThe Q-Code is a standardized collection of three-letter codes used in radio communication. The use of Q-code even in voice transmissions makes communication safer and more efficient. Each set of three letters, always starting with the letter Q, has a specific meaning that is known to the interlocutors.
\nI could see that two of those codes were widely used among police officers in radio communication: QAP and QSL.
\nQAP was used when you wanted to transmit a message. Let's say someone wanted to talk to me, so he said on the radio: \"Bruno, QAP?\". This could be translated as \"Bruno, are you ready to hear a message?\"
\nMy answer could be QAP or for example QRM. If I responded with QAP, I was signaling that yes, I was ready to receive the message. If you responded with QRM, I would be informing you that at that moment the communication is being interfered with.
\nDuring communication, the second most used Q-Code appears, QSL. At the end of each message, the sender ends it with the QSL. This means that he is asking the receiver to confirm whether they received the message. It would be something like: \"There is a suspicious vehicle heading towards your team, QSL?\" If I had understood the entire message, I should respond with a simple \"QSL\". In the absence of a response, the sender repeats the message until receiving confirmation from the receiver.
\nEach message is concluded with a QSL, and its reception is confirmed with a QSL. Does this remind you of any transmission control protocol out here, used on the internet?
\nThe main thing here is that a large amount of knowledge is condensed into three letters, in a language that provides interlocutors with efficiency and security. Eric Evans, the author of \"Domain-Driven Design: Tackling Complexity in the Heart of Software\", would agree with me that there is a principle here that should also be adopted in software development.
\nA language specific to the domain, which considers possible interference from the environment, a ubiquitous language that is known by all interlocutors, can be the difference between life and death, between success and damnation of a project.
\nWhen a project does not have its own language for that domain, this creates the need for numerous translations between interlocutors. Developers have to translate for domain experts. These, in turn, have to translate their demands to developers. Even among developers, there may be a need for translations.
\nAll these translations make concepts confusing and the consequences of a lack of mutual understanding can be disastrous.
\nThe importance of this in trying to arrest a Bug goes much further than compilation errors or exceptions. It goes through the swampy characteristic that the software may be logically working almost perfectly, but it is not delivering what was expected by the stakeholder.
\nLet's look at the following pieces of code:
\nif (order.status != completed) {\n // (...)\n}if (order.isFullyRefunded()) {\n // (...)\n}Here we have a small example of two ways of expressing the same situation. In the second piece of code, a common language between developers and domain experts was used. Here, possible logical errors can be easily noticed, without the need to keep in mind what the possible states of the Order are and what the expected state is for that case.
\nSee the following code:
\nPayment newPayment = payment.add(10);What do you expect from this code? Does it add ten dollars to the payment? Ten transactions, inbound or outbound? Will the method create a new payment object or will it just change the value of the payment variable and create a new reference called newPayment?
\nA lesson also taken from the book \"Clean Code: A HandBook of Agile Software Craftsmanship\" shows that clear communication must also take place between software developers. Functions must communicate exactly what they are doing.
\nIf you have to look at the function's implementation to know what it's actually doing, that's a big sign that you need to rename it.
\nI'm very grateful that you've made it this far, you've certainly learned a lot about police activity and how some of its principles can be applied to combat and remove Bugs from your software.
\nBy applying this knowledge you will be able to work more confidently in your code, giving you peace in your daily life and also allowing you to experiment and discover new things.
\nYour aim will be improved and you will get straight to the point, avoiding damage instead of first looking for the culprits. You become bolder and go further because you know you can trust your tools. This leaves you more time to refine your code and seek excellence.
\nFinally, not only will your code be safe and efficient, but its result will also be in line with what your customers expect because through effective communication you can accurately deliver what was demanded from you.
\nThe result of all this is that Bug is in jail, lives are no longer in danger, and you can go home at the end of your shift to enjoy the rest of the day with your family which is looking forward to welcoming their hero back.
","fields":{"slug":"/bug-police-officer/","tags":["debugging","testing","TDD","DDD"]}}},{"node":{"id":"a34b3781-aea3-55c9-b730-6e611a223ff2","frontmatter":{"category":"Social","title":"Back to Social Life at AUTO1 Tech","date":"2023-08-25","summary":"A hybrid social life has begun in the spring of 2023 after the pandemic. We have experts at AUTO1 Tech that have been invited to different public speeches.","thumbnail":null,"authorName":"Yi-Wen Fang","authorDescription":"Executive Assistant Tech","authorAvatar":null,"headerImage":null},"html":"A hybrid social life has begun in the spring of 2023 after the pandemic. We have experts at AUTO1 Tech that have been invited to different public speeches.
\nOn May 25th 2023, our Director of Software Quality Assurance, Alexander Gyulai, gave a session on the topic of MODULAR AUTOMATION at TEST AUTOMATION DAYS 2023, where networking with software testing professionals and learning from the world’s leading test automation experts.
\nHe focused on topics of scaling, architecture and collaboration in automation testing. And the details covered in
\nHow and why to move common components into modules and how to re-use it across the company.
\nHow did we scale testing of our platform with 700+ services and hundreds of releases per month.
\nSetup release and distribution process of the automation modules
\nIntegrate platform modules into automation components
\nBesides technical advantages, he also shared how this approach improved collaboration and communication between developers and quality assurance engineers.
\nFabian Schomm-von Auenmüller, Head of Data Platform, was interviewed in July by the Data For Good Podcast, which is hosted by Zuma, a recruitment agency focusing on data technology professionals in Berlin.
\nIn this interview, Fabian focused on the monitoring and assurance of data quality within data pipelines. He explained that data observability encompasses processes from logging to monitoring, alerting, and ensuring data quality throughout its lifecycle. Fabian emphasized that catching data issues early is crucial for maintaining trust and preventing downstream problems. He recommended initiating data observability practices as early as possible and discussed tools like Elementary that integrate well with DBT (Data Build Tool). He highlighted the benefits of having explicit tests to validate assumptions and being prepared for failures. Fabian anticipates that future trends in data observability might involve leveraging AI and machine learning for pattern recognition and anomaly detection, ultimately enhancing the quality and reliability of data pipelines.
\n\nWe are proud of Alexander and Fabian and their works that provided an insight to the everyday projects and challenges the Technology team at the AUTO1 Group manages. We hope you find these as interesting as we do. We all grow along with the company, as does our data and infrastructure. Let’s work SMARTER instead of harder!
","fields":{"slug":"/back-to-social-life-at-auto1-tech/","tags":["auto1","engineering","social","qa","dataengineer"]}}},{"node":{"id":"74d57990-d6af-5953-b038-ee5d823d3e7a","frontmatter":{"category":"DevOps","title":"Enhancing AUTO1 Cloud Security Posture with Terraform Security Scanners","date":"2023-04-28","summary":"This article describes our journey to adopt KICS terraform scanner into our toolset. We will go through our approach to choosing the right tool from a vast selection of great options.","thumbnail":{"relativePath":"pages/terraform-security-scanners/logo.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsSAAALEgHS3X78AAABBElEQVQY053RP2vCQBjH8eggRToIXYriVkFUsF7E6CAoaiLdLEJLh7rXLv4Dh0wuQqFQEBxEyFyo+iZcJVC7uPWd6PcpcXAqePDhnuTufsndaR3TbcPBL3aYd033qWQ8+LRzGgELAsb0Q9jU0465mchYRqUTutJflVI1WGgijjyi0Gk5BKmLQgIHeMMLPrqWuyXwWQKzygizZsVEE194xx3uEUMbKW88iZAENrHmz0qoUxPoFo874Os2CFYTVHGFClp4RMR7vsW1BAawRI/AERwJapQH2t7ea7JFAi/pb+Q9vdQFWYy0V1/AQPZ4jkF8Ytazvv8uo2/9+CKp0/Mm2P/fnRwAs2dyCkXlBpwAAAAASUVORK5CYII=","width":640,"height":189,"src":"/static/83bade24f8196b98a976b4fdbaebd2eb/0a508/logo.png","srcSet":"/static/83bade24f8196b98a976b4fdbaebd2eb/0a508/logo.png 1x"}}},"authorName":"Artem Marusov","authorDescription":"Artem Marusov is a Senior DevSecOps Engineer in the DevOps Team based in the Berlin office.","authorAvatar":{"relativePath":"pages/terraform-security-scanners/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEkklEQVQ4yzVTCVNTVxR+v6EznUpYzPISCI1BZQoqLoMFiyyiLKKtUKAFEVpNaiG2yNIqlAwqboAFAgwOGBhQOtRCKsxQQKqY5eVlee9l33cgpGwDndILpTNn7tw753zn+85yobd/iiXvEESqUCnUuJowGMwERnT//CTrdOKxg6yEmKispJOtD5pMRovFZNPgWkyFo4hKKpGL30mhuTdiqViukO8gdcaJ0Zd5Gek0UjCVRIJDSMwwUnxURDybUVrwOQiwmKwEplErMLlMIXkng8RzUkSKqhQ4yOp0uB7V1UaHfXCIGhJLDjodRc/+KCIhkvwxmx4Dh5SXXbZbHXqtAWQBZFIxAoEEQMZ/tF6395vSksL46N/rv5L3NRl/68GETaKGa7U5iUeZlAhK2MwfM067C5CrFBgiQyGZGFGgmBbXWowWn3ehilP2rLp49XX/hmpyTTG5jog2Z4XuF/d+KDgbvjdsdmbW4/ZqCZ1aua0ckknkaiWu1+ptZtvi4hK/slxYU2L7td069jQw99I7NfimrW7qAa+uMO1kbDSOazwur15nBEpRRAkBdtBAoNlmsa+urgu7BdW5KZUXkipSDluH7o7dLimI+/Bmxomy5NjMhOMWo8nj8hn1JqBcIVdBgB1TEeDtcLgCi/7RnvaWr3Ma81LbynIIQfUsv0zAyW2/nNZSms5JPiodFs4HVow6I4FpUbkagJUgjclgdjrdS36/dGRA1tvoE7U5f3nsHbqvF9TOj3UuTXSI7nBaijJt0xPexYDJYAKjAW2GUJkSV2vADrgc7sDyinFmXNp3f3VGiHfVPavI77ySYe5t8IkEHdcvNRdlzxsMHu+C2WDeBSvkgJkAYDDkv5bXHFpi4km9SlA58t0lfnY8sKHrWeP1xbfzU0S9XYHAqtPpMRstBL5TswJRbcsGYKd7ccG/tvH3ZH97f0VWc0GSqLz4JffL9qLUbu7ZW8XnHV6/b3HZbndvgzHNjmwEyCZAzS6Ha947/8/W1utXoy2laXMPOcNVBcJvcyb4JTcy4xqqeP6VDbvLZ7W5jNv7/z94t9s2p9fjW1vfcDlseafjbuV90nE1494XiTcuJLBD3xe0Pl7Z2DJYnCaLU68zAz7QaTBnMCpcp9FbLfYlf2BtfdM/7+HmJMF73mPvJbHIJDh4z3FGyGBPp39zS292anQWjcagVuHgR0ASMQJ2zWq2AubpqTfI3Fv5q8EX/GvcjBMHyEFsctCZWBb3XNwdTt7oyIgE1ZmsHpwwoHKVVCyDMCWOSNCBgWEer/bQkVM/8Thof+NYc814a82PuUmc9Dh+0bma/JRu3qdXslMTTqU3PWybfi3GMJ0SVUN3Gh+dScvZty8mknmAxTp4OOZIF79y9ulddKh17MH3ww3c5/zyhtLzFYUX97OjYSoT2NFjCVev8Xr7nkMwzKKQGQx6JDOCDSycsY/FZH+WnnyzJL+BW1JTWnAxOTGaHUUFMBoThNHhSAqZHhpCZTL3QwyYSacxadQI4KPDu+7gEFpwMJVMCQ8NhbcvZDqNGg7TIsBJpTCoFHBnUsjwv61+3idVvgk1AAAAAElFTkSuQmCC","width":50,"height":50,"src":"/static/73b4f6833b9dadf507bf9df0f8322e21/45876/avatar.png","srcSet":"/static/73b4f6833b9dadf507bf9df0f8322e21/45876/avatar.png 1x,\n/static/73b4f6833b9dadf507bf9df0f8322e21/eb85b/avatar.png 1.5x,\n/static/73b4f6833b9dadf507bf9df0f8322e21/4f71c/avatar.png 2x,\n/static/73b4f6833b9dadf507bf9df0f8322e21/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/terraform-security-scanners/header.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAMCAIAAADtbgqsAAAACXBIWXMAAAsSAAALEgHS3X78AAABpUlEQVQoz2PglDBHQhYQxCFhxiEOQlyS5kAEkYKIICtjAHEkgSxzDjCCmMIlacUF5jIJGTIKGrKJmgDVQDTD9JtzipszcIibg5EZjGHOKmooKOMtr57k7J9k75Xs4JMsoebEKmICsgCmH2IEA1wbxDygbbyy1oLyDmJKXrI67gp6XooGPmIqTrwy1sg6IVrAmpGM5JG2ltFxElIx41c0ZhRU45Iz4pQx5JIzlNB04JGyYhczRTaCAcnBUM3imrZsEmrCKkZ13dPW7zikYOTAwCMrpGLJK22NTTMk9ED6TYHOk9R24JTSlNYyv3r/9cOP/5PzyxnYJMQ0bHFohvrZgl3MjFvKUkjVkklMmYFfLrmgpqSpX9HQjoFTUlARGG0gZ0OjCuZnc3joAeXEVF2MHCP1bPx1rP1VjD0kNe0VDT3UzX307cIk1F3ZIJpBUWsJimeINpARYM1AFeqmweomwUBS0yxE2yJM0zxUwzRYwyxEUsMN6DSUqAIawCWJSD1As5mFDEFI2AiEQGwjCJdV1AQ9nsFusIAmAGjSQ06wwNSGEEFLZAAKTmZdNevBHQAAAABJRU5ErkJggg==","width":1280,"height":748,"src":"/static/84f9df6613e6af34b6e63796561794af/26421/header.png","srcSet":"/static/84f9df6613e6af34b6e63796561794af/26421/header.png 1x,\n/static/84f9df6613e6af34b6e63796561794af/34a7a/header.png 1.5x"}}}},"html":"At AUTO1, we rely on Terraform to manage our cloud infrastructure. And of course, we are trying to pay close attention to its security posture. Over time we realized that relying solely on reactive methods, such as AWS Config and Security Hub, to identify and combat misconfigurations in our AWS services wasn't sufficient. It was imperative for us to shift left in our infrastructure development cycle to avoid repeating the same misconfigurations over and over again. In this blog post, we'll be discussing the journey of how we implemented security scans for our Terraform code.
\nIdea would be to introduce some sort of security tool, that would scan all of our changes introduced in the infrastructure codebase and report on security finding. Based on that we could eventually start to prevent merge of PRs that contain some of the most critical misconfigurations. That way we wouldn't have to chase the same Security Hun findings all of the time.
\nTo ensure successful implementation of Terraform security scanning, we needed to address two key questions:
\nAre we fully prepared to incorporate Terraform security scanners into our daily operations, even if it means potential blocks on some of our changes?
\nIf yes, then which one would be the best fit for our requirements?
\nTo answer these questions, we had to formulate specific criteria that the security scanning tool had to meet.
\n| # | \nOur ideal terraform security scanner... | \nImportance | \n
|---|---|---|
| 1 | \nis able to perform various security checks on the terraform code for the majority of important AWS services (including custom checks: encryption, IAM, public resources, etc. | \nCRITICAL | \n
| 2 | \ncan be integrated into our development cycle for different repositories via Github Actions | \nCRITICAL | \n
| 3 | \ncan be run on self-hosted Github runners | \nHIGH | \n
| 4 | \nhas capability to configure different conditions to fail or pass checks (for example severity of the finding) | \nMEDIUM | \n
| 5 | \nhas capability to suppress findings directly inline in code | \nHIGH | \n
| 6 | \nhas good readability of the scan results | \nMEDIUM | \n
| 7 | \ncan be integrated into AWS Security Hub or AWS Inspector (or some other centralized view for analysis, like DefectDojo) | \nLOW | \n
| 8 | \nhas an open source community version | \nHIGH | \n
| 9 | \nhas good documentation, including mitigation recipes for found misconfigurations | \nLOW | \n
| 10 | \nshould take as little time as possible to scan target modules | \nMEDIUM | \n
| 11 | \ncan scan external terraform modules (local or on Github) | \nLOW | \n
| 12 | \ncan scan single files, not only directories | \nLOW | \n
I think all of the criteria are pretty self-explanatory. Assigned 'importance' will come into play later, for now it just should be noted that of course chosen criteria and assigned values are mostly subjective and very team-dependent — for our specific case this outline was close enough.
\nWith goals and criteria of PoC figured out, we had to round up some contenders.
\n![\"Community](\"https://i.postimg.cc/kMLNzY3y/Picture1.png\")
We started by conducting an initial assessment of the available Terraform security scanning tools. Based mostly on community adoption, we compiled a longlist of contenders, which included:
\nHere it is important to know that PoC took place in the second part of 2022, so our observations and conclusions were made based on the versions of these tools that were available at the time.
\nThe initial assessment was rather straightforward. It involved installing each tool, scanning a Terraform module that was part of our infrastructure, and evaluating the scan results. Based on the results of the evaluation (which basically boiled down to briefly looking at the produced report and comparing it to the results generated by other tools), we narrowed down the list to the following contenders:
\nThe ones that didn't make the cut either:
\nwere producing non-satisfactory scan results:
\nAt the same time all the tools that we chose to shortlist looked pretty great. So, now we had to test each of them properly in order to decide which one would win.
\nTo properly evaluate the shortlisted tools, we decided to integrate them into our main infrastructure repository as Github Workflows.
\nThis way we would gather relevant statistics very quickly and at the same time get a taste of the new way of implementing changes.
\nGithub Workflows is an automation tool that is integrated within the Github platform. These workflows allow you to automate a variety of tasks, such as building and testing your code, releasing software updates, and more. Workflows are defined in a specific file format called YAML, which is stored in your Github repository. The workflows are triggered by specific events, such as a code push or a pull request, and they can run a series of actions in response to these events. These actions can be anything from running a set of commands, to integrating with external tools and services.
\nTypical workflow in our case looked roughly like this:
\nname: tfsec\non:\n pull_request:\n\njobs:\n tfsec:\n name: tfsec\n runs-on: ubuntu-latest\n\n steps:\n - name: Checkout\n uses: actions/checkout@v3\n with:\n fetch-depth: 0\n\n - name: Get changed files\n id: changed-files\n uses: tj-actions/changed-files@v35\n with:\n dir_names: true\n\n - name: Install tfsec\n continue-on-error: true\n run: |\n wget -O - -q "$(wget -q https://api.github.com/repos/aquasecurity/tfsec/releases/latest -O - | grep -o -E "https://.+?tfsec-linux-amd64" | head -n1)" > tfsec\n install tfsec /usr/local/bin/\n tfsec --version\n\n - name: Setup SSH Keys and known_hosts\n env:\n SSH_AUTH_SOCK: /tmp/ssh_agent.sock\n continue-on-error: true\n run: |\n ssh-agent -a $SSH_AUTH_SOCK > /dev/null\n ssh-add - <<< "${{ secrets.TERRASCAN_SSH_KEY }}"\n\n - name: tfsec Security Scan\n env:\n SSH_AUTH_SOCK: /tmp/ssh_agent.sock\n TERM: xterm-color\n continue-on-error: true\n run: |\n for directory in ${{ steps.changed-files.outputs.all_changed_files }}; do\n echo "$(tput setaf 2) Currently scanning module: $directory"\n tfsec $directory --soft-fail || true\n done\n\n - name: Post scan message\n continue-on-error: true\n run: |\n cat << EOF\n If security scan found misconfigurations in the code,\n please either try to fix them or create inline ignore comments for the\n appropriate resources.\n\n Format:\n (Place comment before resource code block or in the offending line)\n #tfsec:ignore:<rule>\n\n Rule identificators are specified in the scan results.\n\n More about tfsecs format can be found here:\n https://aquasecurity.github.io/tfsec/v1.27.4/guides/configuration/ignores/\n EOFLet's go over it bit by bit to explain what exactly is going on here:
\nname: tfsec\non:\n pull_request:\n\njobs:\n tfsec:\n name: tfsec\n runs-on: ubuntu-latestIn the beginning we need to define the name of the workflow (\"tfsec\"), its trigger (in our case — only opening or changing PRs) and where it will run. In this case it will run on Github Managed runners, but it is possible to integrate your own runners deployed, for example, in AWS — then we'd have to reference them using a specific name tag.
\nIn this case we defined only one job with multiple steps.
\nHere they are:
\n steps:\n - name: Checkout\n uses: actions/checkout@v3\n with:\n fetch-depth: 0Pretty standard, we use another Github Action (provided by Github in this case) to checkout target repo.
\n - name: Get changed files\n id: changed-files\n uses: tj-actions/changed-files@v35\n with:\n dir_names: trueHere is the attempt to optimize our process. So, obviously when we started this PoC, we already had a pretty sizable terraform repo on our hands. For that reason, it didn't make a lot of sense (and also would be kind of unfair to a person that wants to make some small change to a specific module) to scan the whole repo and report on it each time someone creates a new PR. So here we use another awesome action to limit our scan to only changed modules. dir_names option indicates that we get only names of the directories that hold changed files, not the filenames themselves. This is used here because tfsec expect scanning path to be a single directory. For tools, that can scan separate files, we would disable this option to optimize scope.
\n - name: Install tfsec\n continue-on-error: true\n run: |\n wget -O - -q "$(wget -q https://api.github.com/repos/aquasecurity/tfsec/releases/latest -O - | grep -o -E "https://.+?tfsec-linux-amd64" | head -n1)" > tfsec\n install tfsec /usr/local/bin/\n tfsec --versionThis one is pretty self-explanatory. We need to download and install the latest version of the required tool.
\n - name: Setup SSH Keys and known_hosts\n env:\n SSH_AUTH_SOCK: /tmp/ssh_agent.sock\n continue-on-error: true\n run: |\n ssh-agent -a $SSH_AUTH_SOCK > /dev/null\n ssh-add - <<< "${{ secrets.TERRASCAN_SSH_KEY }}"This one is actually pretty tool and environment specific. It's relevant for tools that can scan modules that are being sourced from Github repos. Here we allow runners to use previously created SSH key, authorized to pull those modules.
\n - name: tfsec Security Scan\n env:\n SSH_AUTH_SOCK: /tmp/ssh_agent.sock\n TERM: xterm-color\n continue-on-error: true\n run: |\n for directory in ${{ steps.changed-files.outputs.all_changed_files }}; do\n echo "$(tput setaf 2) Currently scanning module: $directory"\n tfsec $directory --soft-fail || true\n doneFinally, we can scan code. We use a list of changed directories, found by one the previous steps, and just iterate through, launching the security tool to scan each one.
\nLast part would be just a message to hint at the next possible steps for the engineers. In our case it was just a link to the documentation.
\nOne important thing. You might have noticed that almost in every step we use this option:
\ncontinue-on-error: trueIt was really crucial at this stage to not block work of the engineers under any circumstances. This option ensured that even if some step of the workflow would fail or produce security finding, it wouldn't block the PR.
\nBecause we wanted to compare all three scanners simultaneously to ensure that our observations were always in the same context, we had to construct a workflow for each one and activate them in our repository. Workflows looked roughly the same with small adjustments specific for each tool. After implementation was done it was time for the team members to continue to write code, push PRs and that way generate more and more scan results. For those who were directly responsible for the PoC it was time to perform some tests against formulated success criteria based on accumulated statistics.
\nBased on the accumulated scan results we tried to estimate how well each tool complies with our requirements.
\nBoiled down results are listed below. We tried to make them at least a bit self-explanatory so we could be able to reference them in the future, when someone will inevitably ask: \"Does anybody remember why we chose this scanner over that scanner?\".
\nWe listed previously defined criteria and assigned numeric value to their importance characteristic:
\nThen based on our observations we graded every tool, basically evaluating compliance of each tool to every criterion. Grades multiplied by importance got us numeric results that we could use to objectively compare contenders. Below are the results supplemented by the comments, intended to justify each grade. For better readability you can check out this Google sheet.
\n\nIn the end we chose KICS as our terraform security scanner. Honestly, I believe that we'd have chosen it even if it wouldn't score highest in our internal test rating (but it also helped to reach the decision, of course). Reason for that would be that basically with the same scanning quality as other just these two KICS's killer-features:
\nmade it the most appealing candidate. It is now integrated to all of our terraform repositories (more than a 100 in total). It scans every PR we make, reports on it and blocks them if any of the findings were classified with severity \"high\" (highest in the KICS terminology).
\nAs for the initial goal — to make our infrastructure a bit more secure — I believe KICS has indeed assisted us with that quite well. Sometimes we just fix one small existing or newly introduced misconfiguration and go on with our day. But sometimes we find something that turns into an epic with dozens of tickets that cover hundreds and hundreds of our resources and services. Both scenarios are great and valuable in the end. Additionally, it spreads awareness across all of the DevOps team about available security options of various AWS services and in general more secure ways to use them (at least I really hope so). Taking all this into account, the effort was well worth it and I think we will continue to benefit from it further and further.
\nThank you for reading!
","fields":{"slug":"/terraform-security-scanners/","tags":["security","terraform","DevOps","DevSecOps"]}}},{"node":{"id":"d53c49b4-fba1-5071-b078-79bbf0f23344","frontmatter":{"category":"Engineering","title":"A wild Thread appeared! Multithreaded Testing and Thread-safety","date":"2023-02-24","summary":"Today we'll talk a little about how threads work on computers and how to extract more value from concurrent tasks. For this, I will present techniques on how to develop and especially how to test code that uses multiple threads, avoiding the main pitfalls that involve this theme. Of course, along the way, you'll also learn more about Java's Concurrent package, non-blocking algorithms, immutability, and more. However, we need to start by talking a little bit about butter.","thumbnail":null,"authorName":"Bruno Morais","authorDescription":"Bruno is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/a-wild-thread-appeared-multithreaded-testing-and-thread-safety/avatar.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAQDAQX/xAAXAQEBAQEAAAAAAAAAAAAAAAABAwIE/9oADAMBAAIQAxAAAAGmbSadPSZMPIBLBy1//8QAHBAAAwABBQAAAAAAAAAAAAAAAQIDIQAQERIj/9oACAEBAAEFAqOVEql2047LLDc7A+i5H//EABcRAQEBAQAAAAAAAAAAAAAAAAEAEBL/2gAIAQMBAT8BC5jP/8QAGREAAwADAAAAAAAAAAAAAAAAAAECEBET/9oACAECAQE/AW9HSSsf/8QAGhAAAQUBAAAAAAAAAAAAAAAAAQACEBEhQf/aAAgBAQAGPwLBZRa4VFYt5Jj/xAAcEAADAAIDAQAAAAAAAAAAAAAAAREhURAxYXH/2gAIAQEAAT8ho6G+i0ElVBP0c6AzfMaH3OHLLoTIz//aAAwDAQACAAMAAAAQcC98/8QAGREAAwADAAAAAAAAAAAAAAAAABExEFGB/9oACAEDAQE/EGQbRPcf/8QAGxEAAgIDAQAAAAAAAAAAAAAAAAERMVFhscH/2gAIAQIBAT8QuM2dElucejs//8QAGxABAQEBAAMBAAAAAAAAAAAAAREAITFBYXH/2gAIAQEAAT8QEiRCoPrgpVComQJxnQidLfOE1gCkvzQ9j4TPv80LcHh6dw+UV3//2Q==","width":50,"height":50,"src":"/static/8d7223890dda036e0121799406a35506/d2d31/avatar.jpeg","srcSet":"/static/8d7223890dda036e0121799406a35506/d2d31/avatar.jpeg 1x,\n/static/8d7223890dda036e0121799406a35506/0b804/avatar.jpeg 1.5x,\n/static/8d7223890dda036e0121799406a35506/753c3/avatar.jpeg 2x,\n/static/8d7223890dda036e0121799406a35506/31ca8/avatar.jpeg 3x"}}},"headerImage":null},"html":"Today we'll talk a little about how threads work on computers and how to extract more value from concurrent tasks. For this, I will present techniques on how to develop and especially how to test code that uses multiple threads, avoiding the main pitfalls that involve this theme.
\nOf course, along the way, you'll also learn more about Java's Concurrent package, non-blocking algorithms, immutability, and more. However, we need to start by talking a little bit about butter.
\nButter is a product that is part of everyday life for many people. Present in cookies, cakes, bread, sweets, and snacks, as well as other culinary items that make our mouths water just imagining.
\nButter is not recent; we have historical records of its manufacturing process over 2500 BC. Most often made with cow's milk but it can also be produced with the milk of other mammals such as sheep, goats, and buffalo.
\nSalt has been added to butter since ancient times as a method of preservation, but over time salt has lost this importance as the supply chain has become generally refrigerated. Today, so to speak, salt is added only for taste.
\nSometimes food colorings are added to butter, in Brazil, for example, it is common to use Annatto seeds.
\nAnd one thing I really like to eat is bread and butter. In fact, a delicious slice of bread just out of the oven with a little butter on top, which starts to melt on the hot bread. Oh, how delicious!
\nSadly, my wife happens to like hazelnut cream on top of her bread. And this created a problem with our breakfast, now we have to get two knives dirty to apply the toppings that we like. A knife for the butter, and another knife for the hazelnut cream.
\nHowever, our concern for the environment and its natural resources, which are not unlimited, made us rethink this process.
\nLet's turn this problem into a software. After all, Moore's law has given us processors with more and more cores, and as nowadays virtually all computers have multiple execution cores, a computer must be able to simulate my kitchen.
\nFirst, we need a straightforward Bread that takes 50 milliseconds to be ready (what a blazing fast oven, huh!?)
\npublic class Bread {\n private String name = \"bread\";\n\n public Bread() {\n try {\n TimeUnit.MILLISECONDS.sleep(50);\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }\n\n public void setTopping(String toppingName) {\n this.name += \" with \" + toppingName;\n }\n\n @Override\n public String toString() {\n return this.name;\n }\n}Now we need a naive knife that can apply the topping to our bread.
\npublic class Knife {\n private String topping;\n\n public Knife setTopping(String topping) {\n this.topping = topping;\n return this;\n\n }\n\n public Knife applyToppingTo(Bread bread) {\n if (topping != null) {\n bread.setTopping(topping);\n topping = null;\n }\n return this;\n }\n}We can verify that everything is working using a test. I'm using the AssertJ library, a java assertion library which I do recommend that you have a look on at, as it could enhance your tests and improve readability.
\n@Test\nvoid givenKnifeWhenApplyToppingToThenChangeBreadName() {\n var knife = new Knife();\n var bread = new Bread();\n\n knife.setTopping(\"butter\").applyToppingTo(bread);\n\n assertThat(bread).hasToString(\"bread with butter\");\n}It seems good, lets's see what happens when we are hungry and want 10 pieces of bread made at the same time! For that, we will need a basket where we will put all the cooked bread.
\n@Test\nvoid givenHungryThenCook10BreadsAtTheSameTime() {\n var basket = new ArrayList<Bread>();\n var knife = new Knife();\n for (int i = 0; i < 10; i++) {\n var bread = new Bread();\n knife.setTopping(\"butter\")\n .applyToppingTo(bread);\n basket.add(bread);\n }\n assertThat(basket)\n .isNotEmpty()\n .allSatisfy(bread -> assertThat(bread).hasToString(\"bread with butter\"));\n}Oops!, it worked, but it took too much time. Each bread takes 50 milliseconds to be ready, so now the whole process is taking 500 milliseconds. If we consider the real-world cook time for bread, it could take hours to cook all those pieces of bread like this.
\nI know what to do, we just need more workforce! I will call my wife to help me. As I said, the majority of current computer machines have more than one core, which allows us to work in parallel. Joshua Bloch, in his book Effective Java, warns that we should prefer executors, tasks, and streams to threads, but for simplicity let me demonstrate this problem using threads by hand.
\n@Test\nvoid givenHungryThenCook10BreadsWithMyWife() {\n var basket = new ArrayList<Bread>();\n var knife = new Knife();\n Runnable preparation = () -> {\n var bread = new Bread();\n knife.setTopping(\"butter\")\n .applyToppingTo(bread);\n basket.add(bread);\n };\n for (int i = 0; i < 5; i++) {\n var me = new Thread(preparation);\n var myWife = new Thread(preparation);\n me.start();\n myWife.start();\n }\n assertThat(basket)\n .isNotEmpty()\n .allSatisfy(bread -> assertThat(bread).hasToString(\"bread with butter\"));\n}But wait, the test is failing, what is going wrong?
\njava.lang.AssertionError: \nExpecting actual not to be emptyIt is like the basket is empty, but we didn't change the process at all despite adding the new threads. Yes, it is a common problem that every software engineer faces when starting to work with multiple threads (or multi-thread apps).
\nIn fact, there is a thread that is responsible for running the test suite which executes each step on our test in sequential order. But when we delegate part of the process to other threads, it follows its flow without waiting for them to finish their job. When it reaches the assertion, the previously created threads have not finished creating their bread. That is why our basket is empty.
\nPlease, don't listen to this evil inner thought suggesting that it is just a matter of creating a Thread.sleep() or something like that, so that the assertion waits for the other threads. It will give you a slow and fragile test. On one hand, if you give it a big delay time, it will slow down your tests. On the other hand, with a small delay, the result could vary depending on the processing power of the machine where the tests are being executed.
\nThe answer for cases like those is to use some tools from java's concurrent package. We have to ensure that both workers start to do their job at the same time. As a second requirement, we need that the test thread verifies the assertion only after the job is done by the other threads. For this, we can use a CountDownLatch like those below. CountDownLatch is a synchronization aid that allows one or more threads to wait until a set of operations being performed in other threads completes. It does it by holding the thread until the count reaches 0.
\n@Test\nvoid givenHungryThenCook10BreadsWithMyWife() throws InterruptedException {\n var basket = new ArrayList<Bread>();\n var knife = new Knife();\n var startGate = new CountDownLatch(1);\n var endGate = new CountDownLatch(10);\n Runnable preparation = () -> {\n try {\n startGate.await();\n } catch (InterruptedException e) {\n throw new RuntimeException();\n }\n var bread = new Bread();\n knife.setTopping(\"butter\")\n .applyToppingTo(bread);\n basket.add(bread);\n endGate.countDown();\n };\n for (int i = 0; i < 5; i++) {\n var me = new Thread(preparation);\n var myWife = new Thread(preparation);\n me.start();\n myWife.start();\n }\n\n startGate.countDown();\n endGate.await();\n assertThat(basket)\n .hasSize(10)\n .allSatisfy(bread -> assertThat(bread).hasToString(\"bread with butter\"));\n}The startGate will hold all threads so that they start virtually at the same time when the test execution reaches the startGate.countDown(). After that, the assertion will not be verified before the endGate reaches zero, which will only happen when all threads have finished their tasks.
\nThe bad news is that we have a problem with this current implementation. If you are unlucky enough (or simply repeat this test a few times), you will face the following error:
\njava.lang.AssertionError: \nExpected size:<10> but was:<6> in:\n<[bread with butter,\n bread with butter,\n bread with butter,\n bread with butter,\n bread with butter,\n bread with butter]>It happened because we are using as our basket an ArrayList, which states in its documentation:
\n\"Note that this implementation is not synchronized. If multiple threads access an ArrayList instance concurrently, and at least one of the threads modifies the list structurally, it must be synchronized externally.\"
\nIn other words, this implementation of the List interface is not thread-safe. Once more, to face this issue, we have at our disposal other collection implementations on the concurrent package. Instead of ArrayList, we could use CopyOnWriteArrayList or a ConcurrentLinkedQueue.
\nAs the first one is more efficient in use cases where mutations are less frequent than other operations, let's stick with ConcurrentLinkedQueue.
\nvar basket = new ConcurrentLinkedQueue<>();But as soon as you execute the test repeatedly, you will see errors like those:
\njava.lang.AssertionError: \nExpecting all elements of:\n <[bread with butter,\n bread with butter,\n bread with butter,\n bread with butter,\n bread with null,\n bread with null,\n bread with butter,\n bread with butter,\n bread with butter,\n bread with butter]>\nto satisfy given requirements, but these elements did not:\n\n <bread with null> \nExpecting actual's toString() to return:\n <"bread with butter">\nbut was:\n <bread with null>We should have a problem in another part of the code, could you imagine where it is?
\nThings get worse when my wife starts to make her bread with hazelnut cream:
\n@Test\nvoid givenHungryThenCook10BreadsWithMyWife() throws InterruptedException {\n var basket = new ConcurrentLinkedQueue<>();\n var knife = new Knife();\n var startGate = new CountDownLatch(1);\n var endGate = new CountDownLatch(10);\n\n Function<String, Runnable> preparation = topping -> () -> {\n try {\n startGate.await();\n } catch (InterruptedException e) {\n throw new RuntimeException(e);\n }\n var bread = new Bread();\n knife.setTopping(topping).applyToppingTo(bread);\n basket.add(bread);\n endGate.countDown();\n };\n\n for (int i = 0; i < 5; i++) {\n var me = new Thread(preparation.apply(\"butter\"));\n me.start();\n var myWife = new Thread(preparation.apply(\"hazelnut cream\"));\n myWife.start();\n }\n\n startGate.countDown();\n endGate.await();\n assertThat(basket)\n .hasSize(10)\n .haveExactly(5, new Condition<>(bread -> bread.toString().contains(\"butter\"), \"butter bread\"))\n .haveExactly(5, new Condition<>(bread -> bread.toString().contains(\"hazelnut\"), \"hazelnutcream bread\"));\n}First, notice the use of a new helper method from AssertJ, we specified that the collection should have exactly 5 items that satisfy a condition.
\nThe Condition entity from the AssertJ library helps us to create a whole new set of assertions. If you are interested, please take a look at a delicious example in the appendix of this article.
\nBut let's come back to our bread story. After executing the test now with hazelnut cream, sometimes we face errors that are even strange:
\njava.lang.AssertionError: \nExpecting elements:\n<[bread with hazelnut cream,\n bread with hazelnut cream,\n bread with hazelnut cream,\n bread with hazelnut cream,\n bread with butter,\n bread with hazelnut cream,\n bread with butter,\n bread with butter,\n bread with hazelnut cream,\n bread with hazelnut cream]>\n to have exactly 5 times <butter bread>Why do we have only 3 bread with butter, but 7 bread with hazelnut cream?
\nBefore advancing, let's recap what we have seen until now:
\nMost today's computers have multiple cores, allowing us to get even more value from concurrent tasks;
\nWorking with multiple threads creates some difficulties when we depend on timing to verify our tests' assertions, nothing that couldn't be handled using tools from java's Concurrent package, like CountDownLatch;
\nAssertJ is a java assertion library that helps us to improve our tests also on their readability.
\nWe should keep in mind that not all classes are thread-safe, like ArrayList for example. When many threads will share access to a common collection, we should use one of the thread-safe Collection implementations from the Concurrent package;
\nOk, with all that knowledge in mind let's figure out what is the reason for the weird errors that we are facing.
\nFirst of all, this is a typical symptom of a Race Condition, or more specifically, a Data Race. Quoting Brian Goetz, in the book Java Concurrency in Practice
\n\"A race condition occurs when the correctness of a computation depends on the relative timing or interleaving of multiple threads by the runtime; in other words, when getting the right answer relies on lucky timing.\"
\nOn our baking issue, we have multiple threads fighting to access the same shared property. My wife and I are fighting for toppings at the same knife.
\nIn the real world, one could think of it like this:
\nI get my knife and put butter on that;
\nJust when I leave the knife to take the bread, my wife gets that knife and puts hazelnut cream on it;
\nI take back the knife and, with tears in my eyes, realize that I'm topping my bread with some strange brown cream;
\nNow my wife with her bread in hand stares at an empty knife, trying to figure out who is sabotaging her.
\nNow the fix of our code is just a matter of adding a synchronized block using the knife as the lock. With this, only one thread at a time can take the knife and execute the code inside it. \"Oh! This is the solution for everything, we should synchronize everything!\" Think twice. Synchronization isn't for free; it creates a performance problem, as now all other threads are blocked waiting for the availability of the lock, which in our case is the knife. In our kitchen, we could say that when my wife touches the knife I become blocked, waiting for her. If the phone rings, she should say: \"darling, please answer the phone\". But I will be blocked, I'll not be able to do anything except wait for her to leave the knife. A blocked thread goes to the TIMED_WAITING state, and during this time it can not be used. It is really advisable to keep track of how many threads our applications are using because threads are a scarce resource. But there is light at the end of the tunnel, at least in this concern you should know that Java 19 is coming up with the concept of Virtual threads, in its Project Loom. This new feature will cut the one-to-one relationship between Java threads and OS threads, which means that more than one Java virtual thread will be able to use the same OS thread. With that, when some virtual thread gets blocked by an I/O operation, for example, another virtual thread can perform calculations on that same OS thread. But, as we said, until there we should use our threads with wisdom. \"But, what to do?\", you could ask. Well, if you need better performance you can use Nonblocking Synchronization. Nonblocking algorithms are widely used by JVMs implementations, Operational Systems, and in Java itself to implement locks and other concurrent data structures. Those Nonblocking algorithms are way more complicated to design and implement, but they provide better performance by avoiding the thread block itself and give us other benefits like being immune to Deadlock. To provide a simple example of Deadlock, imagine that my wife and I are in the living room. The remote controller of the TV and the sofa are Locks. To watch TV, a person needs to sit on the sofa and press the button on the remote controller. A Deadlock would occur if I get the remote controller and my wife sits on the sofa. She will be blocked waiting for the remote controller, and I'll be blocked waiting for the sofa. And we will be like that \"until death does us part.\" Let's go back to the Nonblocking algorithm. First of all, we need a way to ensure atomicity to our tasks. And it is possible because modern computer processors have special instructions for managing access to shared data. A common approach is the implementation of the compare-and-swap (CAS) instruction. It guarantees us that the access to the value and its update will occur as a single task. Java's Concurrent package provides us with some so-called atomic classes. Let's take AtomicInteger as an example, its documentation states that it is \"An int value that may be updated atomically\". The following code shows us the same algorithm, but only the last one achieves thread-safe by delegating its thread-safety to an AtomicInteger: And how does AtomicInteger ensure atomicity? Using the CAS algorithm! Its incrementAndGet method executes a code that could be simplified like this: To start, it gets the current value and stores it in a local variable. Then using the compareAndSetInt method it first checks if, at that precise moment, the current value is the same that was earlier fetched. If it is so, it proceeds with the update and then returns True. If the current value was changed by another thread, the compareAndSetInt method returns False and restarts the process. It is worth saying that the compareAndSetInt method is a native method, which uses the most efficient instructions provided by the underlying hardware. With this new knowledge, what about trying it on our bread? Let's remove the blocking synchronization and create the faster bread with butter that a computer could have (and some with hazelnut cream too, for the sake of my wife's happiness). We can do this using an AtomicReference. Of course, we did that after removing the synchronized block from our test. Good, fast, and easy, and without thread blocking! But, please pay close attention to this piece of code. We have some problems there as it is some kind of adaptation of the canonical CAS algorithm. The hard part of building nonblocking algorithms is to isolate the shared memory into an atomic single change. But in our baker issue, we have a two-step action: first, we set the topping to the knife, and then we add the topping to the bread. So we have two states to keep track of: the knife and the bread. As we said, this code works and does not block the thread, but doesn't give us all the advantages of the CAS algorithm, because we still can have a Livelock issue. A Livelock issue can occur even when the thread is not blocked, if it is waiting for an operation that will never succeed. In our code, we used the topping value as a kind of lock. The first thread that moves it from null to another value, it will put all other threads waiting for that value to become null again. It works, but if any thread fails to add the topping to the bread, which moves the topping value back to null, all the other threads will be waiting forever. The control mechanism, when more than one thread fights for the knife topping, is done by retrying again immediately, like in the CAS. But the canonical CAS will not fail indefinitely if any other thread fails. It will just finish its job as soon as no other thread touches the shared memory when it is still working. \"So, what to do!? I'm reading this to find answers, not questions!\" Ok, I'll let you know. The other side of the coin, in the context of thread safety, is Immutability. If we use an immutable object, virtually all problems with atomicity are gone. That's because all problems that we are facing here are based on threads fighting to access the same shared memory. If the memory that they want to access cannot be modified, those risks are gone and thread safety is achieved. We can summarise that an Immutable object is one that can't be changed after its creation. Java introduced the Record, which is a simple way to create immutable objects. We just need to provide the type and name of the fields. The public constructor, equals, hashCode, and toString methods, along with the private, final fields, are all given for free to us, by the compiler. Take a look at the following code: Some could argue that immutable objects are a waste of resources, but it isn't true in the majority of cases. Dealing with immutable objects offers us performance advantages when it frees us from using synchronization or creating defensive copies, it even reduces the impact on generational garbage collection. But talking about defensive copies, is the record really immutable? Look at this test: This task will fail with the following log: We have to keep in mind that in the way that we created the Parent record we have kept a reference to its internals. A record is still unmodifiable, but to achieve immutability you have to avoid exposing its inner state. In our example, we can avoid that by overriding the constructor and coping the list of children's names: Now the test is successful, but we are still publishing the internal state of the record. The getter method for the property childrenNames is publishing the reference to the record's inner state, allowing it to be modified: This code will fail as well: To avoid that you have at least two options: override the getter and return a copy of our list, or the one that I prefer: save our internal state as an unmodifiable list: Like this, every time someone tries to update our inner state, they will get a UnsupportedOperationException, which I believe is what we should expect from trying to modify something supposed to be immutable. What we have seen so far? We learned more about Race Conditions, in real life and in the software, and the main solution tool from Java: the monitor pattern, using the synchronized keyword; After understanding the downside of locking, we were introduced to the compare-and-swap (CAS) nonblocking algorithm, which despite being hard to develop can lead us to better performance and protect us from Deadlock For the cases when we are unable to isolate our shared memory, or even in other scenarios where we just need more protection, we talked about immutability and how immutable objects can grant us thread safety. We then revisited the java Record, which is a fairly easy way to create an immutable object, but that is not immune to bad programming technic. To really achieve immutability in our objects we saw how we should avoid publishing its inner state. Oh, it was an amazing journey, we started with a brief introduction about threads, how to properly test them, and we met some new friends from Java's Concurrency package. With new algorithms and tools under our belt, we are ready to travel to the multithreaded world riding our thread-safe software. I'll now eat my bread with butter, see you soon! Baking delicious cakes with AssertJ's Condition entity: As you may have read about Application Cockpit in one of our previous articles, it is “The tool which lets concerned developers quickly find their bearings and gather relevant information pertaining to a particular service running in our ecosystem”. Over time, AUTO1 engineers and managers grew fond of using Application Cockpit, and started relying on it on a daily basis. The increase in popularity, naturally, led to development of new features, out of which, one of the arguably most impactful ones was the \"dynamic service search\" (or \"App Cockpit Search\", as we call it internally). AUTO1 platform currently consists of 600+ microservices, and on average two new services are created every week. Many times in the past, we needed to search our AUTO1 service inventory for services which were matching some specific criteria, e.g. are written in a specific language, using certain frameworks or libraries, or have certain features enabled. As it happens, this is actually where App Cockpit App Cockpit Search chimes in. From the user’s perspective, it is an UI that lets the user input some search criteria using the Cockpit Query Language (or CQL for short) and be presented with a list of services matching the criteria. One typical use case is searching the AUTO1 service inventory for the usage of a specific library in a specific version. Below is an example search for services using Once we were certain of what we wanted to achieve, i.e.: search our AUTO1 service inventory using dynamically defined criteria, we began the efforts of defining the architecture. As you may recall from our previous article about the Application Cockpit, we use Backstage as our Front-end. Backstage is fetching data from our backend-for-frontend service called “Application Cockpit Data Provider”, which in turn, is backed by a PostgreSQL DB. As Backstage itself comes with search functionality that could be potentially used for our purposes, we were left with two options, either to reuse the already provided solution and tailor it to our needs or create a solution from scratch. After evaluating the Backstage search, which at that time, was still in the early alpha stage, it was clear to us that it would be a bumpy ride. As it is a full-blown search mechanism, or to put it more bluntly, an over-blown mechanism, compared to what we really needed: As we already use Spring Data JPA as the persistence layer to communicate with PostgreSQL in our data provider service, we have limited our options to two choices: dynamically built queries on top of joined database tables or database views. Either of them seemed to us, initially, as a good candidate for dealing with the complexity that stems from the heterogeneous data structures. The idea behind this solution was to create a set of SQL queries, backed by corresponding Spring Data JPA repositories, one for each attribute in question, execute these queries against the DB, and then combine the results on the Java side performing the necessary filtering and aggregation. The advantage of this solution was that we would keep things small and isolated. Each query is in a separate repository. Each one would deal just with the data structures it's aware of. On the other hand, this solution would come with a significant performance penalty, due to the substantial amount of data that would be transferred from the DB, just to be later filtered out on the application side. The alternative solution would be to dynamically build a large SQL query that would perform both the retrieval of the data and the filtering and aggregation of its results. However, this approach would come at an increased maintenance cost. As every data structure change would require a corresponding Java code change, hence we would lose the benefits of having the data values stored in the JSON objects. On top of that, maintaining a large SQL query would quickly become a problem on its own. Knowing the shortcomings of the dynamically built queries, we decided to evaluate another promising approach - the database views, that could hide the complexity of dealing with the heterogeneous data structures: In that approach, we would create a database view that could look like the following (based on our previous example): which returns: The main database view, here called As previously mentioned, since we use Spring Data JPA extensively, the database view would be reflected in the code as a JPA entity which would allow us also to use the Criteria API to build a generic query on top of that main view and utilize the built-in PostgreSQL functions, that would come especially handy when interacting with the JSON objects. As the DB views seemed like a good solution, we still needed to answer one more question: whether to materialize them or not. As materialized views would be, obviously, faster to query, but that would not come for free, as we would need to build a mechanism to frequently refresh the materialized views, which would also induce a performance overhead. In the end, as we expect the App Cockpit Search to happen fairly infrequently and we value accuracy over query execution time, we’ve decided to not materialize the views. Finally, we decided to move forward with the DB view based solution. Although it’s not a silver bullet and requires us to keep an eye on the query performance, as over time the number of sub-views will grow, which might impact the overall search performance, we consider this to be a fair trade-off for the gained extensibility and maintainability of the search functionality. In this blog post I want to emphasize how important the process of exceptional software delivery is. I’m going through all the steps used by AUTO1 to achieve the best possible implementation in software development. All in all, there are 5 steps that require a clean and effective implementation: The Aggregator Pattern is usually a standalone microservice. It offers a uniform API that a client can use to access data from different microservices. This composite service is utilized when we need to prepare the result data produced from the outputs of several microservices. The aggregator pattern is designed to combine data from several endpoints and provide it as a single endpoint to the client. If necessary, it can also cache that data according to predetermined criteria or rules. You can read more about this pattern here: https://www.edureka.co/blog/microservices-design-patterns So to sum up, we can simply execute two or more api requests to separate services and accumulate the data we need to show the results in the list. This sounds good, except for one drawback: Performance! It is at the expense of performance. One more way to enhance this behavior is by using asynchronous api requests, which basically execute these multiple api requests in parallel. However, it does not provide the best performance (depending on the use-case, it was not an option for us). Since the data like VIN, stock number, etc. will not change in the future, importing fixed data to the service database sounds very suitable. So the key message here is: don't rely on sending synchronous queries to obtain data that was previously controlled by another microservice if your primary microservice requires it. Instead, use consistency to duplicate or propagate that data (just the properties you require) into the other service's database. The only data that changes over the course of a refurbishment is the state. For syncing the refurbishment state, we used asynchronous messaging. Every time the initial state of a refurbishment is changed, a message will be produced to SQS (Simple Queue Service). From there, we can easily consume that message and change the state in the new microservice. To provide a fast search, we added an index only to the state field, since too many composite indexes can have degradation on insertion, deletion, and updating performance. The Data Manipulation Language statement has to modify the table data and the indexed data too, which might decrease the performance depending on the amount of data. We have slow query alerts in place, so we get notified when it might be time for an investigation to improve the performance. Event sourcing is a way for an event-driven microservice to respond to events in order to carry out its tasks. A higher-level operation is performed by a number of these services interacting with one another in an event-driven architecture based on messaging. The interaction usually consists of a series of messages taken by the various services to handle, for e.g. state changes. Messaging at AUTO1 works in the following way. Every service that publishes messages must specify the SNS topic to which it intends to deliver those messages. Topics are bound by message type, so if the message type is So to keep the state changes updated, we created a Message Listener that listens to state changes and updates the state. A message producer for the state changes was already in place since several other services needed to consume this message as well and make their own updates. Feature flags can be helpful in a variety of situations. It's essential to understand that feature flags can be used to configure a wide range of application components. Some feature flags are used to run specific parts of the application, and some feature flags intentionally prevent that. Additionally, feature flags may be applied to the frontend and/or backend. Enabling or disabling features for individual parts of an application is a common use case. The strategy we usually apply is to enable features for specific car repair workshops using a feature flag. That way, we can steadily monitor for errors and let the feature evolve so it is ready to be rolled out to all the other car repair workshops as well. For some features, it might be needed to backfill data to pre-existing entities to support the feature in earlier refurbishments. Quality assurance is an essential part of software development and therefore should ideally take place not only at the end, but throughout the development process. If you want to create good software, you should include software tests in the development process from the very beginning. Here are three reasons for the need of good-quality testing: A high-quality user experience is guaranteed by quality assurance Quality assurance saves time and money More security is provided by quality assurance To improve the quality of an application, you need monitoring. Application monitoring helps developers to locate bugs easier and speeds up the development time. If application monitoring is ignored, it could lead to unexpected crashes or poor performance that would cause significant harm to the company. It can enable engineers to gain insights into how their systems will behave. This information enables them to identify issues or flaws while they are still small enough to be handled with relative ease. At AUTO1 every new service has its own Grafana monitoring dashboard to provide several key metrics like: With over 400 locations across Europe, managing our network devices has become a big endeavor. In order to be more efficient on this challenge the Corporate IT Network Team recently implemented Unimus. Unimus does network automation, configuration backups, change management, updates and enables automation for all the processes. This also benefits the engineers not to work late hours or on weekends. To provide an example, here is a quick overview of how we upgrade our MikroTik routers using Unimus, which are located at our Autohero workshops all over Europe: Welcome, fellow concerned developers. In our previous installment, we have promised to delve deeper into the model that Application Cockpit is using underneath to see how it synergizes with the high-level architecture in order to make the whole product simple yet powerful and extensible. Be advised, that while we have always aimed for simplicity, our initial take on the model was a little bit more complicated, but after more-than-a-few heated debacles, we have stripped it down even more and so far we are pretty happy with it, despite more than a few \"I wish we had the initial design in place\" moments of weakness. In honor of the early version, we are also going to take a short trip down the memory lane to see the trade-offs and compromises we have decided to make along the way and features we have decided to add. We are going to discuss the direction we took and what were our reasons to do so (or in other words, \"what were they thinking ?\"). But for now... As we hinted in the previous blog post, Application Cockpit is actually binary, in a sense that its primary domain consists of only two data types: Resources and Readings and the (extremely) short story is that a Resource is an entity for which we are going to track Readings, while a Reading is a piece of information pertaining to a given Resource. The longer story follows below. Like mentioned above, a Resource is mostly something that we are going to collect readings for, it can be a physical entity, or a purely logical one. Examples of currently tracked Resources include: (semi)physical LambdaDeployment RdsInstance ServiceDeployment ServiceInstance logical Lambda Service SoftwareDependency SoftwareDependencyVersion The Resource has a very rigid model, comprising of only a few, well-defined properties, that is: id - the unique UUID type - the type of the resource, see above for examples name - a textual identifier, unique across one type; needs to be deterministically derived from resource's properties available at runtime (further details in tradeoffs chapter) parent_id - id of the parent resource (if any) created_on - a timestamp and a late-comer to the party, but we find it useful in many ways, especially during troubleshooting In order to publish (a.k.a create) a Resource, the collector will provide only the name, type and optionally the parent identifier. When it comes to type, it is a free-form String. We have no predefined enumeration to represent it and the collectors (see previous article to learn more about them) are free to push whatever types they want, akin to the front-end which is allowed to ask for any type it wishes. In addition, the inclusion of parent id allows us to build hierarchies, so that our data closely follows the domain that we are modeling. Given the above, you should now have a general idea of what a Resource is and how we use it, so we can move on to Readings. Since we have established that it is generally a piece of data pertaining to a Resource, let's have a look at its properties: id - the serial identifier; type - type of the Reading; free form String, akin to Resource type resource_id - uuid of the Resource this Reading describes value - free-form json document to hold the data of the Reading storagemode - determines the behavior when multiple Readings are published with same resourceid and type: SNAPSHOT - only the most recent Reading is preserved TIME_SERIES - all of the published Readings are stored Since the values we store for Reading are free-form JSON, it gives developers a lot of flexibility in how to handle their flows. We store both trivial things, e.g.: To the humongous too-big-to-fit-in-the-picture guys: Again, akin to Resource type, the Reading's type and value are completely opaque to most of the Application Cockpit infrastructure, so individual contributors are free to ship anything they wish in those properties. The only components which need to be aware of the exact structure are the Collector that is originally publishing the data and the UI component which is querying the Data Provider in order to render the respective view. That way, many developers are free to add their own custom extensions, without stepping on each other's toes or needing to seek approval or assistance from the core Application Cockpit team, which in addition can remain blissfully unaware of the atrocities that the others may commit and attempt to hide within their custom Readings. Having dealt with the detailed model description, we can now move on to discuss how our journey with its design looked like. Of course we, being the pragmatic, more-than-once-bitten little members of the software engineering club that we are, were under no delusions about what is going to happen to our nifty design when it gets into its first bar fight against the necessities of real life. The only question was when we are going to be faced with our first moral/design dilemma and how much are we going to be willing to compromise our collective conscience. When it comes to the little trade-offs, actually, the more observant reader might have noticed by now that something is a bit amiss in our picture: In this article we have learned that in order to publish a Reading for a Resource, the Collector probably needs to provide the resource_id to bind it to the right entity In the previous article we have mentioned, that Collectors are generally stateless and have no access to the storage holding Resource data In consequence, how can a Collector find out the correct id to publish the reading for ? Well, it would that it could, but it can’t, so it shan’t. It’s quite a bind we find ourselves in, right ? We were adamant about not allowing collectors access to our DB, so we had to find another way to make things happen… A short look at the ReadingPublishMessage model, might provide some hints, though, so let’s start with that: Does anything stand out to you ? It is almost a reflection of a Reading, but with a slight twist, and you probably got it right, so enter Resource's name - that tiny little property looks like it has been put there in order to provide a human-friendly name for a Resource, but it couldn't be further from the truth than that. In fact, initially it was never there (we use Readings to store human-friendly names of the Resources anyway) and we added it specifically to solve the issue at hand. If you recall, the description says that it needs to be unique and possible to deterministically assemble from the data available in the runtime. Leveraging that, the Collector doesn't need to know the right id, it just needs to provide the correct textual name. The good news is that we have a way of overcoming the initial obstacle, but there is also bad news... This forces every collector to be aware of the logic required to construct names of the Resources it is interested in, moreover, the logic needs to be in multiple places and it needs to be synchronized. But let's leave it at that, since data ingestion is going to be the focal point of the next article in series. At this point a meticulous reader with a penchant for domain modeling might begin to wonder: \"All right, a Resource can have a parent... That's nice, but is that really all that there is? You say you want to closely match the domain, but not everything is a child of something, sometimes you might want a Resource to 'own' Resources, so... how about that, smarty pants ?\" This was also something we were intensively brainstorming about and lots and lots of coffee cups later (and probably more than a few beer mugs, too) we have arrived at something we were not very uncomfortable with. Our main priority was to avoid complicating the model (remember ? we want people to integrate easily, so we want to keep things simple), so we didn't want to introduce any new concepts or tables. Instead, we elected to cheat a little, so please have a look at the example below with a picture being worth a thousand words: Basically we have elected to model a \"has a\" relationship between two Resources using an intermediary Reading to store ids of the referenced (one or many) Resources. In our example a Service Deployment has a Reading Called \"Service Deployment Dependencies\". The main purpose of this Reading is to hold data which we can use to construct id of \"Software Dependency Version\" Resources that our \"Service Deployment\" has. As you can see, if we wanted to obtain vulnerabilities of all dependencies a specific Service Deployment Resource has, we could easily construct ids of each and every \"Software Dependency Version\" and then query for its \"Dependency Version Vulnerability\" Readings. There aren't any places where we use this not-so-orthodox approach, but there are some and the example below is the most prominent real use-case. We are closely monitoring the performance impact it has on our service and so far, it has been negligible, or at least negligible enough that we are still able to look at ourselves in the mirror. Having confessed our most prominent sins, let's wrap things up with a less controversial story. There are several items that were not part of the initial design, which we have decided to add along the way, because we have found them useful. The storage mode for a Reading was a last-minute addition to the picture, but it serves its purpose well. Initially we thought of Readings solely as means of representing the current state of things, e.g. the single-point-in-time data like: tech stack given service is using AWS account id of the deployment IP address of an instance, etc. This meant that we were only interested in the most recent value of a specific Reading Type for a Resource, always replacing it whenever a new value was collected. From the more technical standpoint - we could get away with performing upserts on Readings with the key being comprised of resourceid and readingtype. However, after a few initial releases, we've discovered that there are certain pieces of data more linear in their nature, which Application Cockpit users could really appreciate on a daily basis. A prominent example of such an item is Service Events Graph, which as you can probably guess represents a service's life cycle timeline, so a mixed bag of Service Deployments, Instance Startups and Shutdowns, Incidents etc. Here's what it currently looks like: What we have also found out rather quickly was that it would be quite awkward to represent this kind of data using our initial point-in-time-based approach to Readings. Enter storage mode - a property of a Reading which tells the underlying infrastructure how to behave when a new Reading for the same resourceid and readingtype is persisted. To preserve all the already existing flows, we have assumed the default mode to be SNAPSHOT, which upholds the default behavior of upserting new values, keeping all the implemented features intact without any additional effort. As an extension of the functionality, we have introduced TIMESERIES mode, which preserves all the values that were collected and allows us to construct e.g. neat timelines like what you can see in the screenshot above. As a bonus, here's how the actual data underlying the graph looks like: Whilst it seems quite natural for a Reading to have a timestamp since it is collected (or, as you will, measured) at a specific point in time, it's not so straightforward in case of a Resource, especially not from the Business Domain side of things, since it mainly represents a certain Entity that simply is. Seems like we're not all about Business, I guess, but we'll get to that in a while. With the above in mind, the first iterations of the Resource model did not have a good reason to include timestamps, so they didn't. However, as we kept adding new features and collectors, we have started noticing that the system is accreting quite a lot of Resources and that a rather large portion of them is only relevant within a short timeframe. It wasn't a problem for us at that point, but even though we hate doing the house chores as much as we do, why not keep our room tidy and \"let go of the things that don't matter anymore\", as we've all certainly been told at least a few times in our life ? The part where it most stood out for us were Service Instance resources which by their nature are short lived, since as soon as the instance is dead, we don't care about it anymore (apart from audit reasons perhaps). The point was - how do we know whether we can safely delete the Instance Resource ? As you have seen above, we have ServiceEvents telling us when instances were terminated, but can we rely on the event to always be generated and are we willing to trust it ? Not really. Alternatively, we do track instance health checks but they are not ideal by themselves, particularly since a freshly minted Instance would have no checks at all in the beginning and then report as unhealthy for some time before it becomes stable, not to mention that in some cases it is all right for an instance to fail some of the inspections for a while and still survive. The list of things to consider goes on and on, which went contrary to our goal which was to avoid creating a too-sophisticated cleanup-logic for fear it would become too brittle to be effective. The introduction of Resource timestamps allowed us to reach all the objectives in a relatively low-cost and simple manner. Once we started tracking when an Instance was created it was a simple manner scanning Instance Resources on a schedule, discovering those which had no health reports for quite a while and were not created recently to safely trim them away from our storage. The added bonus is that having a Resource creation timestamp comes in handy during investigation of issues (not that there are too many), since it makes it easier to track the timeline and wrap your head around what happened when and why. At this point I am going to conclude my musings and I hope that you've found them interesting or perhaps even inspiring at some point. The topic is far from exhausted, especially given that our journey with Application Cockpit is still well under way and we keep on adding new features into the application, so please look forward to the next article in series. Developing a machine learning pipeline generally requires a custom approach. There is no off the shelf, well-architectured system that will work for all use cases. On the contrary each use case requires you to design the structure that will satisfy your particular requirements. The process for developing, deploying, and continuously improving them is quite complex. There are three main axes that are subject to change: the code itself, the model, and the data. Their behavior can be hard to predict, hard to test, explain and improve. There are some aspects that are important to know before you start building the system. For example: For this example, we will take the use case of making vehicle recommendations for our Newsletter. The newsletter is sent out once per day and will contain a set of recommended vehicles. As additional information, we have a lot of new items added to our inventory each day. Therefore our system will have better performance if we train the model more often. Since the newsletter is daily, we can make a single prediction and have the data ready for the newsletter to use. And of course, rather than perform these steps each day, it makes sense to automate this process and save our time for more exciting challenges. With this in mind, we can start answering some questions and determine which components we need and what the general structure of the pipeline will be. Even with a described use case and defined requirements, there can be many possibilities for design structures. We are extensively using AWS as a cloud provider. So, we will use SageMaker to automate training our model, and as storage for the temporary results, an S3 Bucket. Data preprocessing is a separate step and will be implemented outside of SageMaker. The final consideration is how we will deliver the predictions to other teams. By default SF sends the output of a previous state as the input of the following state. It provides service integration, which means it can connect and use other AWS services as steps. During execution there is an available context object, which contains info about the state machine and execution. This allows the workflows access to information about a specific execution. There is also an execution log, where you can find info about tasks and time spent in each step. This can be helpful for debugging or tuning your execution. This is how all these services were brought together to implement the pipeline using AWS services. We are extracting data from the Data Warehouse to Glue jobs where the preprocessing happens. It then stores intermediate results (as a least effort solution) to S3. The Training job starts as the next step after the Glue job finishes. Keep in mind that the name should be unique within the region and account and it is a required field. In the Step Functions we also specify the S3 location for the output artifacts. The next step is to set up a schedule. For this purpose we created a Cloudwatch event that triggers a state machine and executes the pipeline. The last step in the pipeline is to load output data from S3 to a database that will be accessible through an API. This task is solved using Lambda. It is another AWS service that runs code based on the incoming request or event. In our case Lambda populates predictions from S3 to Redis. Redis is an in-memory structure store that not only supports a caching solution but also has the ability to function as a non-relational database. It will override any specified fields that already exist in the hash, or if it’s new, create it. It works well for our use case. Finally, we created an endpoint using FastAPI that fetches data from Redis and provides it to the other teams. To create all these services manually and deploy them would be time consuming and difficult. Instead we are using Terraform. It is software that creates infrastructure and allows you provision resources quickly and consistently and manage them throughout their lifecycle. By treating infrastructure as code we can avoid costly mistakes that can come from manual setup of services. Here is an example of Terraform instructions that manages schedules in CloudWatch and triggers a Step Functions. To check for any changes: If nothing new was found let’s just abort the pipeline and wait for another CRON or manual trigger.\nIn case a new release was found, we need to build it, deploy it, and run E2E tests.\nIf there are no failures the pipeline will create a new core-test-service release.\nFor those cases, we are using parameterized jobs, for example, to trigger the E2E tests we can use the following commands: The new release will be picked up by the production pipeline by checking the latest tag creation time in our repository: If the release of core-test-service was created within the last 2 hours we will deploy it by triggering another pipeline. As the last step, it will run E2E tests. As mentioned above, both testing environments will use a shared codebase for acceptance testing. It can be done by setting up one extra job that will fetch an external repository using this block: We chose Cucumber as our main test framework since it will give us unparalleled leverage compared to other choices. Tests are written using natural language in a way that allows for troubleshooting to be done basically at a glance. For example here is the scenario that tests the libraries responsible for sending messages in our platform exposed via HTTP endpoint: The above, combined with this generated report, is a joy to work with.\n Furthermore, we decided to create a dedicated slack channel to notify us about the new releases and test results. So after getting this message We can finally get our much-needed stress-free beauty sleep, not that it will help us much, right? Hopefully, this small article will get you in the mood for some nice automation work in your company. Have fun and enjoy your peaceful nights. Let's talk about harmony! If you have a look at an English dictionary, you'll see that even though almost everyone has some knowledge of the meaning of the word harmony, it's not so simple to properly define it. There are many contexts that it may apply, such as people relationships, business, law, and so on. But, at the end of the day, this concept imported from the music means that something happens in a way or frequency that makes it pleasant and less error-prone. So I want you to focus on the idea that the proper frequency is a way to achieve harmony. There are some cases where you would like to limit the frequency that something happens to it work properly. Imagine that you have a coffee machine, it's obvious that you would like it to dispense coffee at a frequency that you can replace the cups and be able to serve all your guests. It happens everywhere, even on software. At least while we have limited resources, we have to orchestrate them to work in harmony. One strategy is to use Rate Limiters, a maneuver to control the frequency in which some action is handled. A common pattern to achieve the Rate Limiter strategy is to use the Token Bucket algorithm. In a nutshell, the Token Bucket mimics a real bucket where Theme Park tickets are stored. To enter to play around at the Theme Park, you need to take a ticket. But if the bucket is empty, you have to wait until some new tickets are put in there. Doing like that, you become able to control the flow of people to the Theme Park by controlling the number of tickets that are put into the bucket and the time interval in which the refill occurs. If we put 10 tickets per minute on that bucket, it'll assure that at most 10 people will enter the Theme Park in each 1-minute interval. Libraries like Google Guava, Resilience4j, and Bucket4j are examples of how we can achieve rate-limiting in our Java software. Despite being a beta version by the time of this writing, using the Google Guava RateLimiter library is pretty straightforward. You create a RateLimiter object defining the number of permits per second and, for each task, you specify that it should \"acquire\" the token that allows it to proceed. For Resilience4J you need to create some configuration before and decorate your task: In the same path, now using Bucket4j, you need to specify the refill interval and the limit: But if you want that the employees from the theme park's main entrance do other tasks while the customers are in the queue and still not allowed to enter, I mean if you want that your thread is used for other tasks, at least these 3 libraries have a pattern that may be a problem to you: they block the thread. Now we are in the second part of this article, and as we are talking about Threads, it's good to know some basics about them. The first step is to know that they allow us to work concurrently, two tasks running virtually at the same time. But Threads are notoriously complex and error-prone, inaugurating a new sorts of problems like Deadlock, Race Condition, and so on. As we said at the start of this article, considering that we have limited resources, we have to use Threads with wisdom and be as clever as possible. If I and my wife are in the kitchen and we have just one knife available, it is appropriate that once one of us stop using the knife we leave it available to the other over the table, so that the other could start working with this piece of cutlery. When one forgets about this rule, the other one is kept waiting for the tool, and important work is not held, preventing the flow of a harmonious cuisine. The computer environment works like this kitchen, we have a limited number of Threads. And when something blocks one thread with a Thread.sleep() or LockSupport.parkNanos(), this Thread goes to the TIMED_WAITING state and during this time it can not be used. It is like the unused knife in the cooker's hand of an unharmonious kitchen. In practical terms, consider the following code: It will output as follow: What we wanted was that our 4 tasks were completed in a frequency of at most 2 for each second, but the Rate Limiter also has blocked the other tasks. You may ask, what to do? Perhaps, we should schedule instead of blocking! Imagine when you need to go to the doctor. Unless there is an emergency, you will likely have to make an appointment. The schedules are spread in a timeline, considering the doctor's ability to do the entire checkup: anamnesis, some additional exams, and so on. So to say, if the doctor takes an average of 30 minutes to attend to a patient appointment, those appointments should be guaranteed in intervals of 30 minutes. With this in mind, we could refactor our previous code. If we need that our tasks are executed at a rate of 2 per second, it means that we should schedule our tasks for every 500 milliseconds, which is one second (1000 milliseconds) divided by 2. Now that we know our interval, we have to find a way to keep track of our \"appointments\", to avoid scheduling more than one task to the same interval. Let's use an AtomicInteger class for that. Let's scratch some code: It will output as follows: Oh, it worked, but seems like we are cheating, because we just added a new thread to the game. How can we know for sure if that new thread is not being blocked? Well, we can test it like this: And voilà: The downside of this approach is that the ScheduledExecutorService, that we receive from the newSingleThreadScheduledExecutor() method of the Executors class, uses an unbounded queue. This means that no task will ever be rejected (by the meaning of the unbounded word, it is like an unlimited queue), and because of that, we have the risk of consuming too much memory if we receive tasks more quickly than we can handle them. Of course, we can avoid that in many ways. One approach could be to track the number of scheduled tasks and reject new tasks if we hit the specified threshold. First of all, let's refactor our code so that we use a queue of tasks as the source to be used by the scheduler. Our log: Now in this version, we are using the scheduleAtFixedRate() method, which frees us from keeping track of the already used intervals. It now checks for new tasks for every expected interval. But what would happen if we have more tasks coming than we are able to handle? The following code creates a new task for each 250ms but handles new tasks just in an interval of 500ms. Our output would be something like this: The size of the queue will not stop raising, leading us to an out-of-memory error. Oh, but don't be so afraid my friend, as we said we can limit the schedule based on that queue. Like if you call the doctor to make an appointment and they answer you: \"sorry, for the next 10 years all our schedules are already taken, please call in another day (or year).\" Our output: Here we have to talk about backpressure. Remember that this problem about memory only occurs if always the input of tasks is faster than the output. If the input of tasks was faster in just a point of time, this surplus of tasks will be kept on the queue and properly handled in the following time. So with this approach, our application could control the rate of events so that fast producers (sending new tasks) does not overwhelm our consumer. In a blocking approach, the producer will be forced to wait. In the scheduling approach, the new tasks will be handled in time, and only if it overloads that some of them will be queued. You could say, but how long should be my queue? Well, I may say that it depends on your use case. A doctor could have a schedule book for all the current year. And if it shows to be not enough, the doctor can buy a new schedule book for the next year. As in your application service, if you can afford more memory, your queues can be as long as you want. But in the case of a traveling circus, if it will be in the town for just 3 weeks it doesn't make sense to accept schedules for the next month. The same as your application, if its capacity of handling tasks isn't fast enough at a point that the handling of an older schedule task starts to become unmeaningful, you should shrink your queue. We can conclude that scheduling is a good approach for rate-limiting without the downside of blocking the thread. It gives us the opportunity of controlling the flow of incoming tasks, queueing peaks, and always handle the tasks in harmony with all other components. I don't have time for automation - how often do you say that? I said that once and now I feel a bit ashamed thinking about it. To never repeat this mistake I had to find some solution, to not only have enough time for automation but also to be much faster using it. I realized that when I was repeating the same test case manually for the 5th time, it was incredibly boring and each try was taking 15 minutes of my time. Today I would like to share my experience with you on how to make it faster and more pleasant. In the big picture, it is all about being agile, splitting the job between small pieces, done as fast as possible. Before the feature is fully ready for tests we have so many different opportunities to start working on automation. An hour of planning can save you 10 hours of doing - Dale Carnegie As soon as test scenarios are prepared, you can select candidates for automation and choose which ones should be executed on API level and which on UI. It should be a part of a well-prepared automation strategy (more about test automation strategy here https://auto1.tech/efficient-testing-1/ ). Check whether all necessary API calls are available for you (maybe you need to ask developers to create new endpoints?), what will be the steps to prepare the environment for the test, explore existing automation infrastructure and recognize how many new methods and classes you will need to prepare. Sometimes our features are settled on some other pages or we need to go through other features or applications to do our tests. As these parts already exist, it is a great time to cover it with automation if nobody hasn't done that before. It is also a great time to consider which tools will be needed and find new libraries that will be more effective. Test data means all that needs to be prepared before your real test will start. When you are sure that you know all the preconditions and steps required to arrange the system before beginning the testing, you can recognize how to prepare it just when the first version of the specification is written. For example, let's take an order in the shop. When developers write their code you can write your own which will prepare test data via API, so following our example the order in specified state. To make it more efficient you can store example of the order in the queue (read more about it here https://auto1.tech/faster-automation-tests/). It will be so helpful when you will need to execute the first most basic test and speeds up the whole process significantly. Developers in your team will be also very grateful for the created test data that they can take and quickly test their changes on. If you use Page Object Pattern you can create a page objects skeleton just now based on the design. Prepare all classes storing variables with empty locators, write all methods which will use these locators to interact on the page and prepare bigger steps that will do your manual work. When the front end part of the feature will be implemented you can just fill in locators one by one look at the DOM and adjust steps easily. With unique locators added by developers, it will be almost a pure pleasure. Let's take this design of the saved searches page as an example and imagine a test scenario that verifies that saved searches are displayed on the proper page and the user can use a saved search to view the current results of searching. We know that to be able to execute this scenario we need the signed-in user so we can prepare this user just now. Looking at this picture we already know that we have an upper menu that should be already covered with automation (if it isn't covered, the time to do it is just now) and we need to use it to open my account view. We have also a side menu to open the saved searches page from my account context. We know how the saved search will look and which button we will need to click to open search results. Having all this information we can prepare the saved search page class, empty locators for all elements and methods that will interact with the page. Many times I saw this approach to automation. I will do everything manually and leave the automation creation at the end or it doesn't make any sense to create automation now, because I will have to change it. Have you seen something similar? But how much time will be needed then to write all automated tests? Two weeks, or more? Who will give this time, when a new urgent feature is upcoming? From my perspective, it is really hard to find that person and it is well-argued. That's why in my humble opinion it is much better to split your time into small sessions before the feature is ready for tests and then have all automation components already done, then just write tests on demand. Isn't it beautiful having valuable tests from the start and be able to use them as regression tests at any time? I will give you my example: I was testing integration between two systems. Requirements were very agile, five teams were engaged in this feature and multiple layers of integration were created, so we had many bugs by the way. I had to repeat every test scenario 10 times on average. Manually test data preparation for each took 10 minutes, the real testing part between 5 and 10 minutes. After the third attempt, I was tired and frustrated. The more tired and frustrated I was - the more mistakes I made. It was a critical point for me to start writing automated tests. Of course, I had to spend a few hours to have it ready, but what a relief I felt when the 15 minutes scenario was done by my automation in 3 minutes. The other big benefit was that automation doesn't make mistakes and I could run my tests how many times I needed so I've stopped caring about the number of repetitions. I was able to share these tests with my colleague that needed them for testing the other part of this integration. My rough calculations show that I saved a few days and so many hairs which I would have ripped out of my head if I had to do all these tests manually to the end. When I was asked about regression I needed only a few minutes of my effort. We have so many opportunities to be well prepared for upcoming features and to reduce the effort needed in manual testing. Creating smart automation will develop our coding skills and help to save time and frustration. I am sure that there are many other ways to receive significant benefits by automating early, we just need to use them. If you are already familiar with MDC (Mapped Diagnostic Context) you can go to the next section of this article. If not, please take a look at this first to get a better understanding of what MDC is. In one of our services we had an endpoint responsible for invoking process related to car auction with a lot of logs for debugging purposes. We already had a request-id to trace it in Kibana, but we wanted to filter logs in a more convenient way, by value that we know even before invoking this endpoint which was the stock number of the car. So we decided to use MDC as it allows us to add diagnostic key-value pairs to every log in given Thread. Although MDC is a great tool, it forces programmers to write clunky code due to the fact that every key-value pair that was set should be cleared, otherwise in some edge cases it could be used by another thread. Clearing can be done by invoking MDC.clear() or by MDC.remove(key). First one clears all MDC keys so we could accidentally remove keys that were set higher in stack. Second one is better then but it has to be invoked for every key. And also in both cases it should be done in finally block to be sure that no matter what happened, context was cleared. MDC also has To stand against MDC drawbacks, we came up with this functional style solution: At first glance we can see that there is no Try Catch block anymore and we are only setting key-value pairs and not clearing them, however everything works exactly the same. So how is it done? Solution is pretty simple. It is based on a container class for storing MDC context keys mixed with Step Builder Pattern. Let's take a look at implementation. LogContextSteps is a wrapping class for two step interfaces: ContextStep which contains method for adding new log key-value pair ExecuteStep which also extends ContextStep and serves execute method responsible for invoking action that we want to wrap with MDC. implements interfaces from stores MDC keys in executes action that we want to wrap with MDC context For testing purposes there are some package protected classes and methods like So, how does this flow work? We begin with the static factory method MDC key is stored in Finally we can invoke Assuming that we have an ELK stack for monitoring and proper log configuration we can go to Kibana to find interesting logs via our MDC key-value pairs. Depending on the log configuration, key can be directly accessible or under the flat field (which was in our case) For key-value pair: stock-number - SN12356 Kibana filter may look like this: Hello everyone, my name is Ihor Sila and I'm a Senior Software Engineer in AUTO1, with over 6 years of developing experience. When I joined AUTO1, I received my laptop and I found out that it's a new MacBook Pro with M1 processor and I believe that's really great, because I've read a lot of articles about this chip's performance and comparisons with Intel i-core series. However when I've tried to build microservices that my team presented to me, I found some unusual situations with tests, which are working with embedded libraries such as redis and postgres. When I've run first time integration-tests that using embedded-postgres library, I've got next exception in stack trace: With the postgres problem, the solution was pretty simple and after I found it in our Slack channel I applied it and it solved the problem. So, when you are setting up your Mac, please check the environment variables and switch locale to en_US.UTF-8 from default one, because the embedded postgres library tries to start its library with a different locale and has problems with the process id. Another solution is to increase the size of a single shared memory segment a Linux process can allocate. So the commands in terminal will be the next And that you should restart your MacBook to apply all settings. After that the problem should gone. Moving on to redis, the problem was that it was newly added to AUTO1. So, from a code perspective everything is fine, but when you try to run the integration tests that are using embedded redis, they just won't be executed, because redis doesn't want to start. There is some exception log that you receive but that's about it, nothing special. When I've tried to debug the library code, it also appeared fine from a first look, but when I've debugged the stack trace of the startup, I've realized that this old version of redis doesn't know anything about the new M1 architecture and that macOS can be built on something different other than an Intel chip. Basically, old redis library only knows that macOS can be either x86 or x86_64. After a thorough investigation, I realized that there is no solution yet and repositories that provide embedded redis solutions haven't updated their libraries so far. The solution to this problem consists of two parts: To solve the first problem, I thought that I could just download the latest redis-server, then get the application file from the library folder and replace the old app file with the new one. Soon after the first try I've realized that my solution will work only for M1 chips, therefore no compatibility with the Intel chipped Macs. I've dived deeper into the redis library to understand how it knows which architecture (x86 or x86_64) it needs to use. The solution was pretty simple, it just executes the command to get system flags and then analyzes these flags to see which version to use. When I executed the same command on my local machine, I realized that on M1 architecture some new flags were added, which represents that the current architecture is written with a new chip. In embedded redis you can tell redis to use another embedded redis application, but not the solution from the library. If you execute next command at the terminal, you will found extra flags for AMD architecture: But if you will execute the same command on Intel-based architecture, you will see another flags, that are specific for Intel architecture: After realizing that, we created a class so that we can tell the library the fact that if the OS is macOS, then it needs to analyze the system flags. If it will find M1-specific flags (which are not present on Intel architecture), then use the latest redis application file which can be executed on macOS with M1(AMD) architecture, otherwise use the existing solution, because it's working on macOS with Intel processor. We also should store this application file in our resource folder, so in that case new developers can just download the source code and the redis application file will be already there. We've tested this approach on our microservices and it's stable. As for now, our core-team already replaced dependency from old embedded-redis to new one and it's working for both Intel and AMD architecture without additional changes. Thanks for reading! Test data management using SHIFT The test cases, scenarios, and strategies are maintained in SHIFT, our internal (and soon to be open-sourced) test case management system. Our QAs make use of a test data tool which is integrated to SHIFT. The 100+ types of test data required by the teams are generated using automation suites that run in Jenkins. This test data is then queued in SHIFT and as a result increased the productivity of our QA team by reducing the effort for manual test data creation. Our director of QA has briefed this here https://auto1.tech/faster-automation-tests/. Seamless Integrations Our CI system, Jenkins, is wired to run these tests against our various QA environments. The integration of SHIFT with Slack, Jenkins, and Jira has helped the teams immensely. The status of Jenkins jobs/sanity suites is notified in Slack. The time taken to debug the issue is hugely reduced since SHIFT displays a precise list of failed tests including the screenshot reference, which helps me to quickly understand the exact point of failure. Once I have identified the issue, I fix it and test it locally, then push the branch to our repository. The process is set in such a way that merge can be done only after review, which gives me more confidence and validation for the proposed solution. This helps me address the issues faster and get the system up and running. This is indeed why the QA team at AUTO1 is one of the key pillars of the Engineering team. My automation testing journey at AUTO1 could not have been this impactful if not for this perfect blend of the best tech stacks and resources, and most importantly, my amazing team. My learning on the job has been very effective and hands-on, and it makes each day filled with new challenges and new learning opportunities that help my work create a real impact. From the very many opportunities that came my way and the deep learning curve it has presented me with, I now realize that - the journey has just begun! Hello fellow concerned developers, or, if you are not yet concerned, while it is not the crux of today’s entry, I hope\nthat by the end of it, you might get interested in becoming one and/or find out how you could do that on your own and\nwhat it actually means for us. Still, the focal point of this post is to fill you in on the architecture of Application\nCockpit (or “App Cockpit”, as we call it in private), the newest tool AUTO1 has brought in for its concerned developers. The “why” However, before dwelling on the “how” too much, let us pay homage to propriety and start with the ‘why’ of the whole\nmatter. The main reason for the emergence of the Cockpit stems from the plethora of tools AUTO1 has adopted to help us\ntrace, measure and monitor our services, not to mention alerting us when things go south. This opulence often means that\nif you are looking for a specific bit of information, whatever the reason, chances are the data is there somewhere, but\nyou need to make a significant effort finding out where to look for it first. To achieve all that, the Cockpit needs to pull data from multiple sources, aggregate it and present it in a comfortable\nand meaningful manner to the users, not to mention being fast and efficient about it, which leads us to the heart of the\nmatter - the architecture of App Cockpit. Finally, the “how”…. When designing App Cockpit, we were striving for what probably every software engineer worth his hide can recite even\nwhen woken up at 4 am in the morning - we wanted the architecture to be “open and easily extensible, yet simple and\nrobust at the same time\", preferably allowing polyglot extensions to let every team in the company contribute and share\nwith others if they came up with a cool feature. In order to achieve this, we have split the Cockpit into 5 logical\ncomponents distributed in 3 different layers, as depicted below, with the descriptions to follow. Data Collector - the component responsible for fetching data from multiple external sources, parsing, aggregating,\ntransforming and sending it on its way to the data sink with a tiny stretch it might be called the ETL of App Cockpit Data Sink - a simplistic component with a one-track mind, whose only job is to receive data in a well-defined\nformat from the Collector, occasionally perform some simple and generic operations on it (like basic validation and\nparent id resolution) and persisting it in the data storage Data Storage - a simple storage that holds the current snapshot of App Cockpit data Data Provider (aka Backend For Frontend) - exposes an interface to serve everything our UI needs, both serving the\ndata “as-is” from the storage as well as applying some additional logic to further aggregate and transform it for\nspecial purposes UI - yer plain old Front-end - by design contains no logic, it only displays the data as provided Sweet, but how about a model? The specifics of the model run outside of our scope, however, getting it wrong could severely hobble the architecture,\nso it is a story in and of itself, which will have to wait its turn to be told in another blog entry. For our purposes,\nwe will only briefly touch upon it, mainly to establish the domain. Truth be told, App Cockpit is pretty much a simpleton, since its world consists of only two kinds of things: Readings\nand Resources and everything it does revolves around the pair. Data Collector - it is the only component for which the logical diagram differs from the physical one, albeit by\ndesign. This stems from the fact that the Collector has been meant as the main extension point that other teams,\npossibly using disparate stacks, can easily plug into when adding new features. By all means, a Collector, as previously\nstated, is any component that fetches data from an arbitrary source, transforms it into Resources and/or Readings and\npushes it to the Data Sink SNS topic. In consequence, when you want to start collecting a new type of data or start collecting it from a different source, you\nhave two options - either add your implementation in one of already existing collector services, or write your own new\none, and the only requirement we superimpose on you is that when you publish to the Data Sink, you follow the format we\nhave specified. At the time this post is being written, we have 3 distinct Data Collector implementations: Main Collector - a general-purpose collector, written in Java and providing data for cross-platform features,\ne.g.: service ownership incidents Docker images used and their vulnerabilities service health and deployments currently running instances code insights (quality, PRs, contributors etc) Java Collector - yet another service written in Java, as the name suggests it collects data strictly related to\nservices written in Java, e.g.: runtime information like JVM and GC being used selective list of JVM parameters passed to the instances egress dependencies for Java services (in this context, the upstream services) PHP/Golang collector - the first contribution coming from outside the original App Cockpit team, written in\nGolang, it is going to provision data for PHP and Golang services, e.g.: egress dependencies vulnerabilities possibly many more to come With a picture being worth a thousand words, let’s have a look at a diagram that gathers all of this in one place: Obviously, the actual process of data collection, as it is currently implemented, is a whole story in and of itself,\ndeserving yet another dedicated blog post, especially since we have elected to make all the collectors stateless, which\nhad its numerous consequences and challenges so keep your eye out for the next entry which is bound to come shortly. Closing words and … we will be back Having said, or rather written all the above, we sincerely hope that we have managed to pique your interest. Not only in\nour solution but also in actually getting “concerned” about your ecosystem, because it does not necessarily have to be\nboring, especially when you have the right tools to make your life easier. On top of that, you can have your fun\nbuilding the actual tools, deftly combining business with pleasure for a change of the usual pace. As hinted numerous times, this is only the preliminary article, meant to lay the groundwork for future additions\nexploring individual pieces in more depth… We are already looking forward to writing those, so keep your eye out! The AUTO1 microservice landscape is growing in size day by day, with new services from a spectrum of languages being erected and usually communicating with each other by using REST contracts. In a fast paced environment, such as AUTO1, often the service providing the REST API and the service consuming this REST API are developed in parallel, by independent teams. On top of that the business requirements over the lifetime of a project evolve, thus impacting the stability of the API. Having these moving parts in picture can make the development process quite challenging, especially when aiming for high productivity, zero downtime and independent development of the client and server parts. Hence, having a clearly documented and language agnostic API is vital for the success of the whole process. After extensive research, we found a great candidate seemingly accommodating all the aforementioned requirements - OpenAPI. In short the OpenAPI specification defines a language agnostic standard allowing to describe REST APIs in a human readable form, imposing a unified structure, clearly and comprehensively aligning expectations between the consumer and provider sides. Below is an example of such an API described using the OpenAPI 3.0 standard: In general, we distinguish two approaches when developing APIs. The code first approach, which is basically about developing the source code fulfilling the business requirements and afterwards, optionally, generating the API specification from it, which for OpenAPI can be done using the swagger annotations. The second approach known as design first, is all about describing the API in a human readable language first, by using for example the OpenAPI specification, and then optionally, based on this artifact generating the source code for server and/or client. Naturally, both approaches come with pros and cons. As previously mentioned, in AUTO1 it often happens that changes that are being introduced in the API on the server side need to be implemented at more or less the same time on the client side. In such a situation, if the code first approach would be practiced, the consumers would need to wait for the server side implementation to be delivered before starting their part, that would be simply inefficient. Here, for the rescue comes the design first approach, where from the moment when the API specification is agreed upon, both server and consumer side can kick-off their side of the implementation, independently. Additionally, this allows to spot misalignments already at a very early stage, and rectify them before the actual implementation from either of the sides is started, hence less expensive to fix. The OpenAPI comes with a rich ecosystem, from UI tools such as Swagger UI allowing to visualize and test ad-hoc the specifications to AWS integration providing a fast lane to expose the specification via AWS API Gateway. Another very helpful set of tools are the code generators, that can generate client side libraries and server side stubs from the OpenAPI definitions. These extendible generators support a variety of languages and frameworks, which lets one move faster, especially in a polyglot microservice environment. For our Spring Java applications we have decided to go for a generator that is very well configurable and easily customizable. It can be executed in various ways such as via cli, maven/gradle plugin, etc. Since our Java projects use maven for the build automation process, the maven openapi generator plugin was the natural choice for us. Despite the vast configuration possibilities offered by the OpenAPI code generators, in order to integrate OpenAPI into our AUTO1 development world seamlessly and make the generated API and models classes adhere to our coding standards we needed to extend the default Spring generator that we’ve decided to go with in first place, to include some AUTO1 specific code structures. To be more productive, concise and spare some boilerplate code we use in AUTO1 the Lombok library, a lot, hence we couldn’t afford not having the Lombok annotations included in the generated classes as well. To follow the OpenAPI generator practices, we have added a configuration property to control enabling and disabling the usage of Lombok annotations: The above setting would result in generating the following annotations in the output classes: Sometimes the need arises to define custom annotations but only on some specific classes, for that purpose we have introduced a new attribute to our OpenAPI specifications called x-extra-annotation, that hints the generator to add additional annotations to the generated code. For example: The above attribute would result in generating the following annotations in the output classes: Generics is a very powerful feature that is available and widely used in the Java language for almost two decades now. Unfortunately, as of the time of writing of the article the OpenAPI does not support out of the box inheritance together with generics. To make it available to our developers we have introduced yet another attribute in the API specification named x-make-parent-generic specifying the genericType sub-attribute that indicates a generic type of the parent class (in this case PageDTO): The above attribute would result in generating the following generic parent in the output classes: As mentioned at the beginning of this article, our Java services are using the Spring Cloud stack with declarative Feign REST clients, originally the Spring Cloud Netflix Feign module was used for that purpose, however, with recent upgrades to newer Spring versions it was replaced by the Spring Cloud OpenFeign module that resulted in changes in the imported packages (import org.springframework.cloud.netflix.feign.* was replaced with import org.springframework.cloud.openfeign.*). So we ended up in a situation where services use either the old Feign clients (located in their respective packages) or the new ones, yet share the same OpenAPI specification. To support both cases we made it configurable for consumers to go with either one: Another shortcoming of the default OpenAPI Spring MVC generator was the lack of support for making the generated classes implement specific interfaces (using the “implements” keyword), as of version 4.3.1 only the “extends” keyword was supported. We have overcome this limitation by introducing a custom attribute x-codegen-implements allowing us to specify a number of interfaces the generated class should implement. The generated output for the above configuration can be found below: Teams often run into the situation where they have to use multiple OpenAPI specifications to generate the necessary contracts to communicate with the upstream services. Unfortunately, the org.openapitools:openapi-generator-maven-plugin requires a separate configuration for each specification, which was a bit too verbose and too cumbersome for us. In order to circumvent this issue we have created our own maven plugin The main aim of it was to reduce the boilerplate configuration, yet keep the flexibility of the original plugin: When dealing with a large number of microservices a common case is that they share common features, hence common models, within a single bounded context. In a typical Code First approach operating in the Java ecosystem this problem would be solved by sharing a common jar which would be then included by the interested parties. We were interested in achieving similar functionality but from the OpenAPI perspective. We achieved this by simply extracting part of the common functionality into separate specification files and referencing them from the individual OpenAPI specifications that would like to reuse these definitions. The challenge in having OpenAPI specifications referencing other OpenAPI specifications was the dependency management and versioning part. It was especially hard from the consumer perspective, since consumers would need to include both specifications in their respective versions (the referencing one and the referenced one) in order for the code generator to succeed. The solution for this was building API bundling, by adding to the specification a meta attribute called x-bundle-apis we could make all referenced definitions to be included in a single final artifact that was then uploaded in the CI pipeline to artifactory. This meant, from the consumer perspective, that they just needed to include a single dependency now, making their life significantly easier. By default, when bundling the API we’ve always included the latest referenced specification. Since we have had cases where we wanted to give more control to the API owners to specify which concrete version of a “common API” they would like to reference, we’ve included another attribute x-bundle-imports which could be used as following: Despite the vast number of improvements we have already made in order to retrofit OpenAPI and its tooling into our existing AUTO1 ecosystem and practices, there is still room for improvements. Just to name a few. Although OpenAPI specification itself is language agnostic, we have language specific generators which in some cases need to be instructed on how to generate the output code, and unfortunately these generator hints often make sense only in context of a specific language. As an example, instructing the generator to make the generated Java classes implement some specific java interface. Since the OpenAPI specifications are enriched directly with these hints, it results in obscuring and cluttering the contract definition. Yet another challenge would be to generate more complex classes resembling the structure as seen when written manually in the times from before the OpenAPI approach was adopted. As presented in the article, OpenAPI is a very powerful tool that enabled our AUTO1 organization to move forward faster in a more structured and unified way in terms of defining and evolving the REST APIs. It’s not a silver bullet, as the design first approach and the OpenAPI specification come with a number of challenges by themselves. However, with the talented engineering force we have we were able to resolve most of them in an efficient manner, yet every organization planning to adopt this tool needs to weigh in the benefits and the shortcomings and make the call for themselves. Hi everyone! This article is the first in a series of AUTO1 testing-related posts, I hope you’ll find the content interesting enough to return for the rest in the series :) To start off, let’s ask one of the most important questions: In AUTO1 the test strategy is a high level, short and concise document which defines: Scope of the testing Risks - areas that potentially can be broken Automation strategy Required test data Security In agile and fast changing environments we usually do not spend that much time on test documentation as in waterfall. However, even in such cases we cannot underestimate the importance of a test strategy. When you are creating a strategy it becomes your guide, a channel for your thoughts - you have a space to stop for a moment to try to understand what should be verified and do it at a very early stage. It is much easier to go back to the feature if you already wrote down your thoughts and you have some kind of summary. Let’s go through our thinking process, step by step. Start with defining what exactly has to be tested. You can ask yourself a few questions: Who is going to use the feature? What apps will be changed/created? On what devices will it be used? What types of testing will be needed? It allows you to better understand the product you are going to test and the environment setup you’ll need. Now think about all the things that can go wrong. Here the questions will be: What apps/areas can be impacted by our change? Are we modifying dependencies of another system? Is my domain knowledge enough to understand a feature? Are there any missing prerequisites which can block testing? Are changes going to be compatible with existing solutions? Answering these questions will help us create a plan in order to mitigate all risks. It can be planning an onboarding session with someone who has knowledge we need, investigating regression testing possibilities for apps or areas in danger or setting up a plan on how to get blockers done. Automation is an important part of our testing process. We think about the approach on the test strategy stage. We create our tests on an UI and API level. When we’ll have the technical specifications, which include details of implementation, then it is time to plan what and when will be automated. We ask questions such as: Where is the specific logic placed? Is it frontend or backend? Do we have existing automation in this area? If yes, do we have to adjust it? Do we have all prerequisites needed to start writing automation? At AUTO1 we have a lot of connections between several areas. It created dependencies which have to be covered in test data preparation. We have to be sure we know the preconditions and steps required to arrange the system before starting testing. Here we can specify if we need additional onboarding or support from other teams. Having answers to the above questions means we are prepared to plan more detailed test cases and we can use our strategy as some kind of a cheatsheet. However, do not blindly follow what you’ve planned - requirements can change. Review your strategy from time to time, to ensure that it is not outdated. Test strategy shouldn’t take much time but it is important for the next step - creating a test plan. It can be a great guide which will help you to structure your initial ideas and be prepared for upcoming changes faster. BACKGROUND This post is in addition to the previous article about plugin development but with a focus on UI based plugin implementations. Sometimes it can be really useful to have an UI and make it possible to interact with it by doing some checkboxes selections or maybe to have some input areas for text. Let's investigate how to develop a simple plugin with input text area as input entry point and some new generated classes, based on input text as a result. As a source code example we will have a look at source code based on my pet project that also was used as a development tool in AUTO1 to generate non GraphQl based POJO classes. USER STORY As an Android developer I need to interact with backend data. It can be usually presented as JSON data or if we can speak about new and modern practices it also can be presented as a GraphQl scheme. GraphQl is a really powerful framework with its own classes generation framework. But let's go back one step and imagine that we have to do some data presentation classes updated for the project without GraphQl support. To do it, we need to find the target class, compare its fields with new JSON fields and update it. It can be easy if we have only one field changed, but for multiple inner fields and classes with its own inner fields it can be too complicated. We can find some online services to convert JSON to POJO and copy/paste output files, but let's try to integrate this technology to our IDE! And our job will be only to copy/paste new JSON to some window input field! Like this one: To make it possible, let's think about our problem and try to do small decomposition into logical steps. Lets formulate our goals by imaging simple user stories: As a user I want to input JSON text into the input field. As a user I want to transform JSON text into POJO files. As a user I want to see the result notification. We will skip JSON to POJO transformation for this article, but we will learn how to develop a plugin with required user interface. Let's think about option number two as about some library that we already have and can use as a part of our small project. TECHNOLOGY STACK Before we will start let's have a look on basis plugin implementation. There are 2 base options to develop new plugin: We will speak about gradle implementation. Gradle is a really powerful framework and it can be really helpful if we want to simplify our building, releasing and testing. The project structure is a common structure for gradle based projects: Most important topic for us will be the plugin.xml file. We already discussed some details in the previous article and we can be focused on some UI related topics. During the plugin development we can use Java/Kotlin and all utility tools that we want like jUnit testing and dependency injection frameworks. In our current example Koin and JUnit can be found. So, now we have and know about project structure, some utility tools, language to write the code and now we are ready to think how it can be possible to create a user interface and connect it with our IDE. Let's remember about our major goals: GOAL 1 - As a user I want to input JSON text into the input field.\n and see how we can implement it. GOAL 1 First of all, let's learn how to connect our plugin with the IDE user interface. As we can see the IDE has a lot of action groups to do something (create new files for example). According to our idea to have some logic that can help us to convert JSON to POJO it would be great to have a new item in the new file list. It will be something like an action “Create new POJO file”. To connect our logic with the IDE by following this path we have to remember about the plugin.xml file that was discussed early. We can use this configuration file to describe our plugin and how it can be used. Let's find the actions section and define new action to show our input field dialog view. As we can see we can use some properties to describe how users will be available to interact with our plugin. We can define a custom icon for our action, define the position inside of the items list (“NewGroup” anchor “last”) and we can define the class to execute when action will be invoked by the user. Custom action class should extend the abstract AnAction class with the implementation of actionPerformed() method. Inside of actionPerformed() method implementation we can add the logic to show a dialog window with the input field for JSON text. To show dialog view we can easily use DialogBuilder from plugin development SDK and Java swing to create a user interface. More information about actions can be found on the official documentation page. GOAL 2 Let's imagine now that we already implemented logic to show the input text dialog with some selectable options to transform this text. We also added some logic of transformation - that can be some library to work with it or maybe our own implementation. And as the result of that transformation some new POJO files were generated added into our project (We can interact directly with the project or we can use project representation, provided by the plugin development SDK). But what about the next steps? We also need to notify the user about the result: it can be a success result notification, or maybe some error case notification. To implement it we can use default tools, provided by plugin development SDK. We can show some information to the user by using popup, notifications or messages. Let's imagine the following case: we have a text input field and we have some logic to convert/transform the text from this input field. If we can speak about JSON to POJO transformation, we have to understand that any random text cannot be converted to POJO correctly. Target text should satisfy some conditions and also should have the correct JSON structure and it means that we have to not only show some notification if text structure is not valid, but we have to block UI and break our transformation flow. Messages can really help us with it! According to the official documentation: The Messages class provides a way to show simple message boxes, input dialogs (modal dialogs with a text field), and chooser dialogs (modal dialogs with a combo box). The function of different methods of the class should be clear from their names. When running on macOS, the message boxes shown by the Messages class use the native UI. Lets see the following code: As we can see we can use the show dialog method, provided by SDK. We can define some dialog parameters, like title, message, button and of course we can define our custom plugin icon. If the image is not defined, standart Java icon or IDE icon would be shown. More information about images support can be found here. Also as it was already discussed here we can add support for internationalization instead of hardcoding only english string values. And as the result we can see this amazing small UI blocking window with our message text: It looks now that we found a solution to cover the error cases. Now we can prevent users from wrong input and show notification about it. Let's think about the success use case. Let's imagine that the input text was correct and we processed our JSON to POJO transformation as expected. Maybe some new files were generated, maybe it was only one new file. It will be a really good user experience if we can find a possibility to inform our users about it. Maybe one of the possible ways can be to show notification messages. It can be a small popup view with some text, like this one: and maybe we also can print some information to the event log: It looks now that we are ready to totally satisfy our user story requirements! Let's do the final steps and implement the following code: As we can see we can use the create notification method, provided by SDK. Unlike dialog related logic, we also have to define the project that we want to use to show the notification. As an example let's get only the first project (most popular use case), but we have to understand that users can open not only one project in the IDE and we have to take care about it too. So, as we can see it is also possible to define message text. Also we can define the message type (it can be INFORMATION, WARNING or ERROR) and we have to take care about the notification group that was defined inside of our plugin.xml file: FEW WORDS AT THE END As we can see it was not really a big deal to develop some small and hopefully useful tools for daily use or to help your teammates. The major idea is just to understand that integration with IDE can be really simple and when we will finish with it, we can be focused on the functionality of our tool. Perhaps in the future when we will have a stable version with some really cool features we can share our tool not only with teammates that we are working with directly, but maybe it also can be helpful for some developers from some random parts of the world. And maybe together we can simplify our daily job! Big organisations are hard to manage. Everyone knows that. It is even harder to manage if an organisation wants to keep innovating and remain profitable and functional. How do we do that without creating a lot of bureaucratic roadblocks and wrap every single decision in a web of workflows and processes? And what data do we need to know that we are moving in the right direction with the right speed? Can we get this data with the least effort possible? In this article, I would like to show AUTO1 Technology’s attempt to address these questions. With the original approach, we had a few issues on our hands. AUTO1 Technology split to 8 Business Units, each of them has its own set of functions, for example, under the Internal BU, we have DevOps, Design, Core etc. In total, we have 30+ functions and each of them has their roadmap to deliver. And, each team also will need to keep an eye on their technical debt. Every day in A1 Technology we plan new features, define the specs, test and do tens of releases to Production. Our roadmaps are living documents and change all the time as we need to react to market events or predict customers needs. We are a Product organisation, meaning that we split our teams into Product-oriented pods, where cross-functional engineers paired with a Product Manager and a QA. Sometimes, depending on the type of feature, they are joined by a Designer. Historically, to keep the roadmaps up to date, PMs needed to maintain headcounts in a Google Sheet. Consider this scenario - a PM knows that FE engineer John Doe works on a Feature A and their QA Jane Doe works on a Feature B. They update the spreadsheet and then find out that the TL switch Jane to a Feature B 5 mins ago. Now, multiplying this scenario to 30, we will see that the final allocations were not always actual, softly speaking. We could resolve it by telling our engineers to log their time somewhere, and they will need to log every hour of their workday. This way we would see them go to another company very shortly. Engineers like to do what they were trained for and not fill up neverending timesheets. And let’s not forget about the PMs, who were people chasers instead of being a customer/revenue/idea chasers. Finally, they spent too much time on getting these numbers instead of focusing on their daily tasks. Looking at the above issue we can also clearly see another one. To manage all of that we use Google spreadsheet UI and it has its limitations: It’s very easy to accidentally remove a value from a cell and don’t even notice it. Which happened a few times... There are no backups and it's very hard to restore some of the values there. The history of updates is hidden and it's not always working as we need to. The purpose of the spreadsheet is to be... well, a spreadsheet - to calculate something to present some existing data in a table view. This doesn’t work well for our use case. Managing pipelines is much more complex tasks that requires a lot more to it - see the next section. Google spreadsheet also doesn’t allow to track the feature lifecycle very well and validate if some steps were missed. NB: The above sentence is not 100% true. But it would require a lot of complex configuration and a lot of maintenance time to implement such functionality there. Which won’t make it a very robust solution. Our process is quite flexible and depends on the type of the feature and the Business unit but in general consist of some common steps: Requirements gathering Writing the specs Getting signoffs from the SHs Demoing the increment to them etc. Using just the spreadsheet functionality it’s very easy to miss some of them accidentally or to take some shortcuts intentionally. There is no easy way to validate whether all the important steps were taken. Some of our features require designs or mobile app counterpart development. How can we track it more efficiently? In the original approach, we would need to manually duplicate records from the BU’s pipeline to a corresponding Design and Mobile team pipelines. There was no easy way to automate it. So, how one would go about managing all these features across the Tech organisation? To help us with that we came up with TechOS, an AUTO1 Technology Operating System, the tool that would help us with all of that. TechOS designed to tackle the issues above. It allowed us to: Get up-to-date data on our Engineers allocations Free up our PMs from a burden of chasing people Introduce some insurance in a form of backups and changelogs Add a feature lifecycle validation to all the items on the roadmap Automate some of the tasks and jobs Get this data - after all, we are a data-driven company - that we need to back-up our decisions, like: What are the late features?\nWhich of our MVPs were too long and too costly?\nDo we maintain a 1 to 3 ratio of QAs and Devs that we claim we do? PMs create tasks in the UI that we built in house. All our engineers receive a notification that reminds them to submit an update for today. This action only takes up to 30 seconds and it can be done from a mobile phone. We only ask them to choose a task that they were working today from the list of tasks. They choose a task as Primary or Secondary (non-mandatory) and hit submit. We don’t ask about comments or any notes. We don’t ask about the exact time they spent on it. No. Just two dropdowns, one of them is not required at all. We don’t use these updates to measure employees performance. They are only used to know the allocations for the task: On the screenshot above we see a pipeline view with some key data about the Roadmap item and the number of people who are working on these tasks. The decimal numbers are there because we show a weekly average. The best thing about TechOS is its data and how it can be used. With people linked to tasks, we can have very powerful reports that can give us insights into how we work and what needs to change. Here are a few examples. We have a textual or a visual representation of how our PMs, Devs and QA are allocated on the task This report shows us how much effort is spent in different areas of the task: in planning, writing specs, development etc. The above two reports combined are a powerful tool and enable us to be agile and adapt rapidly. No more accidental deletes of data in cells Permissions that allow different access to different areas of the roadmap A lot of flexibility configuring tasks workflows. A bit more on that below: On the diagram above you can see an example of how we can configure a life cycle of a task. Something that wasn’t available to us before. Automation - now we can automatically create tasks in corresponding pipelines if a checkbox was ticked. I.e. if someone creates a feature that requires the Design team’s effort, they can just choose the “Design” checkbox and a corresponding task will be created in the Design team pipeline. In order to scale up our CI/CD build platform running with Jenkins[1], we used to leverage the AWS Elastic FileSystem, also known as EFS, to provide a network filesystem that can be mounted across one or more VM instances. This deployment setup works well with a containerised workload as the orchestrator can schedule the container or virtually any node on the cluster, therefore we need to make sure the storage is available for every instance where the build platform will be deployed. We accomplished this by simply attaching the EFS storage in all nodes on the cluster, thus allowing the container to be rescheduled in any instance at any time. However, we started to face some challenges with the increase of new services and the introduction of features such as Jenkins Pipeline and Multibranch Pipelines. We noticed that the system build queue was growing bigger over time, as our build system was taking longer to process the services pipelines, with a direct impact on the development time and life cycle. In our investigation and analysis we identified the storage file system as one of the biggest contributors to the performance of the Jenkins Master node. We tested and decided to migrate the Jenkins filesystem to another storage type - AWS Elastic Block Store service, also known as EBS, that provided better performance in our tests compared to the EFS file system. The Jenkins Home directory will often be composed of lots of small to medium size files, which can be troublesome when operating in a network file system due to the overhead required to send the file operations over the network. Moving Jenkins to a faster storage system can lead to a great performance impact on the overall build platform. The downside of running the platform on AWS EBS is that we degraded the high availability of the platform as EBS is limited to a single Availability Zone (actually, one of the reasons why the storage is faster), leading to an adjustment to the service deployment, ensuring it’s deployed on the same AWS availability zone and introducing new plugins that will integrate the container system with EBS. The new adjustments also introduce more complexity to the infrastructure. Nevertheless, we decided that improving the performance of the system will bring more value to the development side and consequently could have a positive impact on the business. Most of the performance gain is bound to the Jenkins Master itself, as only the master was using the EFS storage and the Jenkins Build Nodes were already using EBS or instance local storage. Nevertheless, there could still be improvements as the Jenkins Master was responding faster even when when communicating with the build nodes. After the migration to EBS we started to see an improvement on the Jenkins Master Build Queue. In the following graph, we can see the number of builds queued to be processed. The bigger the queue, the longer the waiting time for the new jobs to start the services pipelines build and deployments. On the other hand, with the EBS storage, the build dropped to 4 seconds as shown on the image below: We calculated a simple estimation and we expected an impact of 2 hours on engineering - based on the time Jenkins takes to process the pipeline and start the builds, and the time it takes to scan a multibranch pipeline. This estimation does not account for indirect performance gains such as navigating into Jenkins UI, like when waiting on Jenkins to render the build’s output. [1] Multibranch Pipeline job is processed by cloning down the GitHub repository, scanning on the branches looking for a Jenkinsfile into the project root directory and creating or starting a build on the given branch when a new branch is created or the branch received any git changes. Hello, colleagues, friends, readers and all of you excellent people out there! It's been a while, right? Let's pick up where we left off... Our second Tech Talk took place in February at Maze, an event space fashioned after the Berliner spirit. Antonella and Alexander attracted almost 100 people with their promise of delivering amazing content, while having two different tracks allowed for the exposure of the participants to other vital parts of a tech start-up/unicorn. This event, the second in a series of events meant to promote local growth and accelerated learning, is now available for everyone to enjoy, on Youtube, along with the slides from the presentations. Antonella Zagaria - Localisation in an ever-growing startup\n\n Alexander Gyulai - SHIFT your testing gear\n\n As always, sign up on our community page on Meetup.com to stay up to date and be amongst the first to find out when our next event is going to be. And it might be sooner than you think ;) Intro In this article, we will take a look at spring-batch and how it could be used. We will walk through various configurations and we will create an application which reads from a CSV file & writes into a database with outstanding performance. I used the following: Java 11, Spring 5+, Spring Boot 2 and Maven-based project. Create a Project First, we need to create a Spring Boot 2 project. We recommend that you do so by visiting spring initializer, which is a useful tool to generate spring projects with required dependencies and configurations. Dependencies We need the dependencies below to run and test the project: spring-boot-starter-batch dependency includes all the configurations to run the spring batch application. Lombok is just a helper dependency to write the code faster and cleaner. H2 is used as an in-memory database. spring-boot-starter-test and spring-batch-test are included for test purposes. Book Class Let’s create a model class book, which will represent a book. This will just serve as a model class and will help us during implementation of spring batch. Configuration Let’s create a class called SpringBatchConfiguration. We will add all required configurations here. First, let’s annotate this class with @Configuration to be able to inject beans, and with @EnableBatchProcessing to enable spring batch processing. Additionally, we can add @RequiredArgsConstructor from Lombok which would help us to generate constructor with parameters. These parameters are the ones which are marked as final class properties. These properties will be injected as spring beans. Our class will be like this: Now, in SpringBatchConfiguration class let’s add properties which need to be injected into the constructor: jobBuilderFactory and stepBuilderFactory are declared in spring batch jar as spring beans so that we can inject them in any class we want. File Reader Now, we need to declare and initialize spring beans to configure the batch process. The first bean which we will need will be responsible for reading from a file line by line. Spring Batch provides a default class for it. The class name is FlatFileItemReader. Similarly, spring has different default reader classes for reading from a relational database, mongodb and etc. However, if you need, you can create your own reader and implement it in a way you want. Let’s now see what FlatFileItemReader injection will look like. We are configuring that the bean reader should read data from the given file path, which should be a CSV file having rows with title, description and author respectively. The StepScope means to initialize this bean after each step, so file path could be dynamically set when the spring batch job is launched. We will come to that later, how to pass the file path later in this article. Item Writer Now, let’s create a writer class, which will take the data and write into the relational database. In the writer bean, we are setting item processors and adding values inside preparedStatement accordingly, which book property should be inserted for each db table column. Step Configuration Now, let’s configure a step which will be executed in the batch process. In our case, the configuration will look like this: In our scenario, we have only one step, but it's also possible to configure multiple steps. Here, we are creating a spring batch step with the name step1 and setting reader and writer accordingly. These are the readers and writers which we created as spring beans earlier. We are setting chunk as 100, which means the items chunk proceeded is 100. We can make this configurable too. The processor is for converting the current object to the one which the writer should proceed with. In our case, it is the same object. Job Configuration Last but not least, let’s configure the job which should be executed. Here, we create a job with the name bookReaderJob. We add an incrementer RunIdIncrementer, which is an id generator for the tables which are specifically designed to store the details about job executions in the database. After each time the spring batch job is executed, it will save details about the execution. To check the schema structure for storing this data, take a look at this sql. It is for MySql, but other SQL scripts are available too. Additionally, we add flow by which the job should be executed. Currently, we have only one step in our flow, so we add it. Please also add this config: Execution To execute the job, we need to declare JobLauncher and launch the job. To do so, we need to create the below class: Now, we can call the execute method from whenever we want, with the file path from which the data should be read. Additional Classes There are a few additional classes which you will need to execute your code. Create a class BookEntity so that spring data JPA will automatically create the book table for you. Then you can create repository. Then create the interface BookRepository and extend it from JpaRepository. At the moment, we will need this only for testing. Testing No one likes a code which is not tested. So, let’s write a few test cases for our class. Here is a code sample for test cases: The second test executes in ≈ 3500 ms in a machine with a 2.2 Ghz Intel Core i7 and 16GB ram. It proceeds 60K lines of CSV file, and saves it into a relational database. You can check out the working sample code in github, Back in November 2019, we had the pleasure of hearing from our Chief Product Officer, Niv Liran, about the makings of a data-driven unicorn. This event, the first in a series of events meant to promote local growth and accelerated learning, is now available as a video, along with the slides from the presentation. \n As always, sign up on our community page on Meetup.com to stay up to date and be amongst the first to find out when our next event is going to be. During analysis of why some services take a long time to build, several problems have been found, mostly related to number of Spring contexts that were created during integration tests. More about those problems can be found in previous blog post. In order to decrease the likelihood that slow tests will appear at AUTO1, we implemented an IntelliJ plugin which warns a developer about possible problems when he writes code and provides quick fixes if they are applicable. During the plugin development, we realized that resources on writing IntelliJ plugins are scarce, and therefore decided to create a step by step tutorial on how to implement In our tutorial we will be using the community edition of IntelliJ 2018.3.5. Final source code is available on github.\nThere are several ways on how to create IntelliJ plugin. The recommended one is to use Intellij plugin for Gradle and that’s what we will use in this tutorial. Let’s start with creating new project using File > New > Project..., select Gradle and make sure that in Additional Libraries and Frameworks both Java and Intellij Platform Plugin are selected.\nIf you cannot see “Gradle” or “Intellij Platform Plugin” then please make sure that both “Gradle” and “Plugin DevKit” IntelliJ plugins are installed.\nContinue next steps of the wizard using default values, use any groupId and artifactId. After some time IntelliJ will download dependencies and create an empty plugin. Our inspection should warn a developer each time there is a class annotated with There are two kinds of inspections: Local inspections are executed in the background when file is opened, in general they have access to currently open file. Local inspection cannot report problem for not currently processed file. Inspection class has to extend We don’t need to access complete graph of references and we would like to give a hint that something is wrong as soon as possible so local inspection is a better choice for our use case. First problem that we encounter is to pick proper base class. One approach is to ask IntelliJ to show class Hierarchy of Create new class named The most interesting thing happens in a visitor which is notified each time a Java annotation is encountered in source code. By overriding different methods, plugin can process methods, classes, imports etc. In our case we check if fully qualified name of annotation matches our expectations and register a problem when that’s the case. Later on we will change this class to include also a quick fix. Creation of inspection class is not enough to make it available in IntelliJ - it’s also needed to register inspection in It’s possible to register each inspection one by one in Create the following class to implement our inspection provider: And register it in Now it’s time to check our inspection in action by running runIde gradle task. This could be done from terminal by executing Many inspections report not only problems but also provide automatic ways of fixing issues. In case of DirtiesContext, it’s not possible to provide safe way of removing it because DirtiesContext is often used when bean holds some state which makes tests dependent on each other. Usually we want to remove the annotation and then figure out the \"dirty parts\" and clean them up in an elegant way. Since the second part is hard to automate we will provide quick fix which only removes annotation. Go back to Next step is to pass quick fix when problem is registered: Executing Up to this point we have used hardcoded strings inside inspection name and description. To allow the plugin to be accessible in different languages we can externalize the messages. For this purpose we can use properties file. First create Then create a new file matching Register bundle in And finally use bundle in inspection, for example: We picked First problem was that our tests couldn’t see classes from JDK, which was fixed by specifying project descriptor to use internal JDK: Second problem was that visitor received incorrect fully qualified class name of the annotation. Instead of When dependency is resolved and downloaded into local Maven cache it can be added as library with code listed below: It’s important to use Next surprise is that by default test searches for test data in strange location inside IntelliJ home folder which can be fixed with overriding Inspection tests can be done by providing file to analyse and resulting file that should be created after given quick fix has been applied. Inspection tests use Complete source code can be found at github github If you would like to share your plugin with other developers, you can publish to JetBrains plugin repository as described in the documentation. Other simple option is to execute buildPlugin Gradle task which will create plugin zip file inside build/distributions and then install it via Install Plugin from disk... available inside Preferences > Plugins (in IntelliJ 2018.3 is available through \"gears icon\"). When working on your own ideas you might run into a situation when you don't know how to implement some functionality. In this situation you could try to find the answer using links provided in section below. What also worked for us was reading source code of inspections available as part of community edition of IntelliJ, often there is an existing inspection which does a similar thing to what you might want to do. Devoxx, formerly known as JavaPolis, is an international community conference which first took place in 2001 in Belgium. Since then it has developed into one of the largest Java events in Europe, with chapters in Belgium, France, UK, Poland and Ukraine, as well as an established conference in Morocco. Among the speakers at DevoxxUA (Devoxx Ukraine) 2019 I’d like to mention Josh Long, the Spring Developer Advocate at Pivotal and Cay Horstmann (yes, that’s him, the author of “Core Java”, “Big Java” and other great books). I got a chance to participate as a speaker on the second day of the conference, delivering a presentation on one of our latest solutions for implementing GraphQL with authorization in a microservice architecture. Was very excited to see that the talk got the highest amount of participants for that time slot and the room was packed, including people standing on their feet for the entire duration of my presentation! All in all, it was a great experience of sharing what & how we’ve built while getting valuable feedback and interesting questions. Have to say it felt incredible to stand on stage next to all the stars of the engineering world, during the closing ceremony of the conference! Here's my entire talk:\n\n For the second time this year, members of the AUTO1 Group Tech department came together to code, develop exciting projects and have fun! Disclaimer: in an AUTO1 Group hackathon, the focus is not on delivering as many projects as possible - on the contrary, it is about the fun of developing and having joint team sessions in a creative and relaxed setting. Many thanks to our video crew who did a great job at capturing the atmosphere and collecting impressions, just take a look: \n The overall theme of the autumn hackathon was SHEEP IT, a pun which should need no explanation. A total of 15 teams competed, with around 40 participants in our Berlin HQ. In addition, there were also colleagues from Kiev, Milan and Katowice that joined remotely. The winning team, codename “Bike Thieves”, won by developing an anti-theft Slack bot, which ensures that no bicycles can be stolen in the parking space of our Bergmannstraße HQ. When using this Slack bot, you can protect yourself against bicycle theft as you will be informed by the bot and can quickly check and see what’s going on if your bicycle is moved. As most great ideas, one of our colleagues came up with this during a rather emotionally charged moment for a Berliner - his own bike was stolen during work hours. There is still a lot that needs to be done before this bot can be used in everyday life. Even under the time constraints of the hackathon, team members Sergey, Rafael and Dmitri made substantial progress on their Smartlock. The team was able to convince the other participants of their idea and their problem-solving strategy, which won them the first place. Some bike owners are already looking forward to using the bot. Slack bots were a very popular topic at our autumn hackathon, with two other teams doing a language bot and a book exchange bot, while other projects included various internal quality-of-life upgrades to existing workflows, a browser based car game that was rigged to be unbeatable and a somewhat crazy face&object recognition software that was able to tell us, via Slack of course, how many free spots were in a kitchen and how many fruits were there. In addition to fame and glory, the winners received smart home starter kits, and all participants received a beautifully crafted sheep as a reminder of the hackathon. What’s next? The next AUTO1 Group Hackathon will take place in spring 2020 and we look forward to even more exciting projects! Watch this space for updates, wink wink. For the first time, insights about how we built Europe’s leading digital automotive platform will be shared - including exclusive information about how AUTO1 Group uses technology to build solutions that take the stress out of financing, documenting, evaluating, transporting, and trading used cars. How did we establish automotive expertise, create data-driven decision-making processes, and hire a diverse, multicultural team of car and technology enthusiasts? What kind of challenges did we face when scaling such a business both vertically and horizontally? Questions such as the above will be answered by Niv Liran, Chief Product Officer, who is also going to share with everyone his most important learnings from his time at AUTO1 Group. After Niv’s talk, we will hold a quick Q&A session moderated by Paul Hofmann, Director of Engineering at AUTO1 Group. To end the evening, we're inviting engineers, product owners, and tech enthusiasts to share their knowledge and curiosities with each other over drinks. The event will take place at Maze, a venue near Mehringdamm, Berlin, on Tuesday, November 26th. AGENDA [18:30 - 19:00] Open Doors & Drinks [19:00 - 19:10] Welcome by Paul Hofmann [19:10 - 19:40] Input: Niv Liran [19:40 - 20:00] Moderated Q&A by Paul Hofmann [20:00 - 21:30] Networking & Drinks To sign up for our first Tech Talk and to get to know us better, please register on our Eventbrite page. Connect with us: The PostgreSQL team announced recently a new release of the most advanced\nopen source relational database - PostgreSQL 12. As usual it comes with an impressive list of improvements\n(generated columns ♥️), one of them being long awaited by dozens of developers: improved Common Table Expressions. I should first explain what are the Common Table Expressions for those who are unfamiliar with them:\nthe CTE’s, often called Essentially CTE is an additional query which results can be referenced in the subsequent CTE’s or the main query before which\nit is being placed. It should be clear enough with an example - here’s a sample query with two CTE’s taken from Postgres docs: Now we know what they are, but what purpose can they serve us? Well - we could parry here and say: for the same purpose as ordinary database views serve.\nThat’s of course a dramatic simplification - CTE’s are much more powerful and should not be treated as a simple database views.\nNevertheless I would like to keep the collation for the sake of this article. Database views are absolutely optional - one can simply substitute them with a subquery and achieve identical results.\nIndeed that’s what modern database engines do these days - once a database view is being used they inline it’s query as a subquery.\nWhy to bother then? We are able to deal completely without database views and even if we did use one the database engine would get rid of it anyway. What benefits do views give us then? And why the heck are they inlined? The most appropriate explanation here is that database views help us achieve better readability.\nThey offer an elegant way of abstracting some parts of a database into a meaningful object, often matching closely with the domain. Instead of creating giant and ugly looking queries it is possible to extract some of its parts into an appealing view which is easier to browse and select data from. \"Divide and conquer\" rule in it’s true form. Still, we didn’t answer the fact that the underlying view’s query is most of the times inlined while it is being referenced. The reason is performance of course. Smart guys found out that lazy evaluation helps the optimizer a lot - by delaying the execution we could take advantage of a context of the actual query. This in turn allows many clever optimization techniques, like: pushing down predicates (WHERE filters), eliminating unnecessary JOINS, accessing only subset of columns etc.\nIn other words - a database is smart enough to do as little work as possible when evaluating database views, in the context of the issued query. Personally I love this pattern: aggregating all the data into views and letting a database to optimize my queries - these folks are really good in it and my queries are dead simple too. I have mentioned the CTE’s at the beginning, saying they are able to create temporal data views. I still conform to the comparison with database views - they both are in many cases similar.\nThe main difference is that CTE results are temporary and are reachable only in the context of a query which CTE is being part of. It may make sense to use a CTE in a place where a regular database view is not justified (e.g. it makes sense only in the context of a query and not in the whole domain), expecting similar behavior. Besides many remarkable advantages there’s at least one disadvantage which disqualifies CTE's for most use cases - before PostgreSQL 12 it was implemented as an optimization fence. What does it mean? Easy to imagine an example with a generic data view aggregating lots of data. If the view is then used to select just few records it could mean a tremendous waste of computation, if the aggregation is executed immediately.\nInstead the aggregation should be executed only on a small subset of data, which can be deduced from the outer query. Unfortunately such counter intuitive behavior was true for CTE’s for a long time - their results were materialized only to be accessible for the rest of the query afterwards. Not to blame anybody - the creators had quite good reasons to implement such behavior (i.e. guarantee of exactly one evaluation, possibility of a recursive CTE’s and more) but this still feels like focusing on corner cases instead of optimizing the happy path.\nThat’s why the community insisted for a long time for changing the status quo by giving the possibility of disabling the fence and unlocking the full potential of CTE’s. The SQL gods listen to their prayers and here it is - PostgreSQL 12 with updated Common Table Expressions.\nBy default, when few constraints are met, the queries will be inlined allowing joint optimizations. It is still possible to force the old behavior - by defining the CTE This is truly a game changer for many developers who care about their queries’ readability, allowing them to substitute not very well liked subqueries with elegant Don’t get me wrong - not every subquery should be immediately replaced, they still have their strengths and in some cases they should be chosen over CTE’s. It is just convenient to have two distinct tools in a toolbox, isn’t it? RESTful API’s are ruling the industry nowadays. They are very often associated with an extremely popular data format - JSON.\nIt is so popular that database creators decided to implement it directly in the database engines.\nAll modern databases offer JSON processing capabilities.\nLet’s have a look and see how can we make use of them to improve our application’s performance. The typical choice for a Java application is to use JPA as the persistence layer abstraction.\nIt became the de-facto standard for a reason - it offers great flexibility and ease of use.\nUnfortunately it doesn't come without a price since we have to pay with degraded performance. It is a widely known fact that ORM’s impact performance, compared to pure JDBC.\nOn the other hand, relatively few are willing to use plain JDBC, considering the amount of additional work it requires. Let me demonstrate how to use JDBC to leverage the performance, with the help of modern Postgres features. I have prepared a demo application running on Spring Boot which exposes example data from the northwind database.\nThe data are exposed in three different ways. The baseline version simply exposes JPA entities: The Order entity aggregates data from numerous database tables: Thanks to the magic of Spring the entity is being translated into a JSON response: The demo project includes the p6spy library which gives an insight into the queries being issued against database.\nTurns out that three queries are required to load a particular Order - this means three database round trips. Of course we can force the ORM to join all the tables and fetch them in one hop, but this seems a not so elegant solution -\nit pushes the ORM to load duplicated data and the necessity of deduplicating them in-memory. Can we do better? JDBC is always a choice if the solution could be expressed via SQL query. Luckily we can create one which matches our case perfectly: The query looks verbose, perhaps a little enigmatic.\nIt is a bit long and therefore it could be overwhelming.\nHowever, on a closer look we could notice it isn't so much complicated - there are few idioms which make the query powerful. The good news is that the query generates the exact same JSON as the JPA endpoint provides - it is possible thanks to Postgres’ excellent support for JSON format.\nOur case is not sophisticated though - single One can notice how the nesting has been realized: how to nest JSON objects and JSON arrays, e.g. the The database query could be adapted according to the caller needs - the response can be shrinked, making the query simpler and - most probably - even more effective. Worth noticing is that the join table The query is flexible enough to substitute all JPA features we need. I want also emphasize the fact of CTE’s being used to decompose the query into logical parts.\nPersonally I find it elegant because it is easy on the eyes and it “scales” - meaning while our case gets more complicated (JSON aggregates more data, becomes deeply nested) our query gets longer instead of getting deeper.\nIf the query grows too long we can simplify it by extracting some it’s parts as database views. The CTE has one more advantage here: it gives a control over the keys visible in end resulting JSON ( So we have a nice query which allows us to fetch required data in one hop in a format ready to be consumed by our clients.\nThis means no adjustments are required, therefore the Java implementation can be that easy: We didn’t use plain JDBC here but a Of course we could achieve similar results using plain JDBC (e.g. by using Some folks could argue “I don’t want the JSON to be directly returned - I definitely need the POJO to make some processing over it”.\nThat's an absolutely fair statement! We can satisfy this requirement as well. Let’s point it one more time: our query produces the exact same JSON as the Order JPA entity has been translated into. This means we can use the resulting JSON and unmarshall it into an object of class This version is very similar to V2 but it introduces an additional step of unmarshalling.\nIt is also a great verification step which makes us sure the V1 and V2 results are identical. So how are all the solutions performing?\nWe suspect the V2 version to be faster than the competitors.\nWe are also unsure about the V3 which requires additional unmarshalling phase - could it impact the performance badly? The demo project includes a JMeter configuration and a bash script which verifies the configurations.\nThe benchmark is always run against a fresh setup (db + app).\nThe app is running on java 11 with EpsilonGC turned on - this allows us to verify memory consumption of each solution. The requests are run in strictly defined sequence, fetching the orders one by one.\nThere’s a prewarming phase which issues 100 requests and then a profiling phase with 20k of requests. I have run the benchmark several times on a Macbook Pro 2018, the results are similar on each run.\nHere’s a summary of the most recent run: We can see the V2 outperforming the V1 by 32 seconds - that’s 37% improvement. We can notice also a lower memory consumption on the V2 - it used 52 MB (49 MiB) less memory than V1\n(noted the initial app’s memory usage which is 785 MB): Worth noticing: the V2 version does not leverage Hibernate at all therefore we could minimize the memory footprint even more by excluding the JPA dependencies. Let’s have a look at response times: Once again we are able to notice improved throughput as well as lower latency. The plot confirms that as well (cropped for brevity): Personally I am very well satisfied with the V3 results - the memory consumption being at the same level as the JPA it still offers satisfying performance, nearly as good as the V1.\nIt proves that the unmarshalling does not impact the performance as badly as we might expect. I really encourage readers try running the benchmark and share their own results! Last words about the database: the results are reproducible currently only on Postgres 12,\ndue to recent improvements in Common Table Expressions. I have had a run on Postgres 11 and the V2’s performance was… horrible - I had to shorten the benchmark 10 times to make it finish in a reasonable time.\nHere are the results I got: You can give it a try - the demo app is prepared to allow a quick switch to Postgres 11. The query we have prepared enables fine control over the data being fetched from database. We are no longer restricted to tabular data - we could easily fetch complex data graphs, even manipulate them to match the desired format.\nWe did it keeping the solution maintainable and very effective in terms of throughput. It is still unknown to me what is the main source of latency in the JPA version: either is it the additional round trips or the internals of Hibernate.\nI see the topic interesting to explore.\nPerhaps some day we’ll get an ORM which is able to load entity graphs using a single query.\nAny Hibernate devs reading here? UI-tests are a necessary part of our development routine in AUTO1. Although that wasn’t the case a while ago. One of the issues, which was blocking developers from writing and running UI-tests, was the need to set up UI-tests. Of course, to test the screen in isolation you need to have some architecture, employ the SOLID principles, especially Dependency inversion. With proper architecture it’s easy to substitute dependencies inside screen under the test with fakes or mocks. But even with good architecture you need to provide these mock implementations. And it’s not always obvious how to make your DI framework to provide mocks for tests. Recently we started a brand new project from scratch. And we decided to give Koin a try. Why Koin and not Dagger? — Just because Koin offers concise and minimalistic syntax and it uses Kotlin. Also, important for us was the question of code generation. Although Google announced that in new versions of Android Gradle plug-in they made kapt incremental, we still didn’t want to add this extra burden to our build process. Simultaneously Koin 2.0 was announced at the same time. That’s how we found ourselves facing question: how to provide mock dependencies to our UI-tests? Inside our fragments we inject number of dependencies, typically they include viewmodels: That’s exactly what we want to substitute with mocks. To do so we need a custom Application class, which will start Koin with our test module, not the real one. We start our Koin instance inside But we need to somehow start our test application class. We cannot use Then we need to change our And don’t forget to disable animations for UI-tests: We are almost done. We only need to write our test and provide our test module to it. We are using our custom fragment test rule, which provides a couple of methods for submitting your fragment and mock test module. But you could use the same approach for activities. I am using default Android Finally, our test class could look like this: Here we are using Kakao, it’s a nice tool to simplify working with Espresso tests. It provides implementation of Page Object pattern and allows you to abstract interactions with your UI behind the abstract Screen objects. Using Kakao you can improve maintainability of your tests, so if you are not using it yet, give it a try. Also we are using Mockk for mocking, which uses all the power Kotlin gives us. That’s how with Koin, Kakao and Mockk you could easily test your fragments in isolation, making your life easier. You can check the source code for the sample here. In the last few months I’ve been working a lot with forms. I had to do a large refactoring in one of AUTO1’s applications, and through all the research, I also encountered the term subform. The concept was troublesome not only for me.\nI found more than one issue on the Formik repository about developers asking for help.\nIn this article I'll try to clarify this concept and show you how to use it properly. If you’ve ever worked on a medium-large project with a reasonable amount of forms, you might have probably noticed that you could potentially reuse a lot of fields, sometimes even entire forms!\nThat’s the core of the subform concept: reusable components (fields or group of fields) that can be composed together to create bigger forms. At the beginning I had a lot of questions in my mind like: These questions and many others appeared while I was refactoring existing codebase. Many similar questions created by other developers appeared in the Formik repository as issues. Don’t get me wrong: implementing scalable validation for a single form is not that hard. The complicated thing is to keep validation and state flexible while you write your subforms. What does it mean? We’ll see that in a moment. Formik is an amazing library, one of the bests for this job, and here’s what it brings to the table: Formik provides a Field component that, thanks to the React Context API, will be automatically connected to the Form component that wraps our Field, regardless of how deep our Field is in the tree. Yup is a library to create validation schemas. This library is very similar to the prop-types of React, so it is very simple to start using it.\nFormik supports Yup natively, so you just need to define the schema and pass it to Formik, it will take care of the rest. So this is how Formik will make our life easier not only while creating subforms but also when we’ll need to maintain them! Let’s create our first subform! Before we start, we need to clarify what our subform will be responsible for: In our case, we're going to create a form that allows to create a new user.\nWe'll need to display the following fields: Just keep it simple. So, let’s create our We’ve just defined our validation schema regardless the markup of our form. This file just contains all the validation logic and has one responsibility. If in the future we’ll need to add some more validation options, we’ll just need to change it here. Now it’s time for the default values. Initial values are required from Formik because it uses controlled inputs. So, if you don’t specify those values, you’ll get and error from React as soon as you try to change the content of the input. And now the master piece: The React component. Remember: we just need to use the Fields and not the Formik or the Form components. And that’s it. In this phase we can test every single part of our subform: validation, the default values schema and the React component. A piece of advice: Formik sets the fields’ values in its state using the This mechanism gives us power to improve the flexibility of our subform. How? Let’s edit our component in a way that it will accept an optional property called We don’t need to do that with the other parts of the subform, the main form will be responsible for that. And, about the main form, let’s see how to implement it! Finally, we’re going to create our main form component. Let’s define its responsibilities just like we did with our subform. The main form will be responsible for: It's a lot of responsibilities, and that’s all right. The main form represents a specific point in the UI/UX where the user needs to insert some data. In our case, it could be a registration form, but it could also be a registration combined with a purchase, just like when you buy something from amazon and you agree to sign up in the process. The point is: A Form is a unique component that represent a specific use case, so it has to be designed accordingly. That’s why it makes no sense to create a “god-form” component with hundreds of props that decides which endpoint the form is going to use. It just creates useless complexity. In my opinion, the best way to organize this approach is to create a folder where you’ll store all your subforms. Every subform will be represented by its directory and it will contain all its parts: validation, values, fields and the React component.\nA main form, instead, should be created ad-hoc to fit the needs of a certain use case, for example inside a certain route. So, with that in mind, let’s proceed to the implementation. We’ll have our directory called Fields Names Validation Default values The React component And that’s it! Of course, this is a very simple example, you could have different needs. I want to leave you with some advices that helped me while I was refactoring my codebase. It's good to have them in mind to ease the process of code refactoring. A subform should have only first level values in its state, which means that, when you design a subform, you shouldn’t get crazy about the shape of its values. It should be a flat object and every key should contain the field value.\nThis way it’s so much easier to write validations, default values and error handling (and why not, also the React component). You can avoid this advice only when you are using a subform into you subform. For example, let’s say you have an address subform. It has a lot of fields and a complex validation schema. In that scenario all the logic will be handled by the address subform and you’ll just need to orchestrate it in your own subform, just like you would do in the main form. I didn’t do it in this article but the idea is to export a function instead of a schema. This function will accept parameters that will define the schema that you’ll get.\nIn this case you can toggle the “required” validation in some cases, or other kinds of validation. Example: let’s say that we want to make the “lastName” field optional, but not always. That’s how you could define your schema: Now you have a scalable validation schema.\nIn this way you can always decide if a certain field is required or not. Also, you can extend that function to add parameters. In this way, if your subform scales, you’ll just need to add parameters and adjust the code in the subform accordingly, but every single form that uses your subform won’t be affected by these changes because everything is retro compatible. Dealing with forms is not easy. Of course, there are simple cases, but there are also trickier ones. This was my way to organize the code, and of course, it’s not perfect.\nI’m sure there are other thousands amazing ways to solve this problem better than this, but for now, this is the best way I’ve found to keep everything testable, maintainable and scalable. I hope this will help you, and if you have a better approach, I’m looking forward to reading it!\nUntil next time, happy hacking! Since AWS lambda supports custom runtimes you can use PHP to create AWS lambda functions.\nBut it comes with two major drawbacks With bref.sh these problems will disappear. Let's try to run Symfony framework as a serverless app. What does it mean that something is Serverless? Does it mean there are no servers at all?\nOf course not. There are still servers behind scenes, but you don't have to worry about maintaining the servers and reserve their capacity. Your infrastructure is scaled automatically, and you just pay only for what you use.\nDoes this kind of architecture have only pros or there are some drawbacks?\nLet's leave the discussion of all the smaller pros and cons of serverless applications for another article.\nI will focus on the biggest drawback which is an impact on our code. Frameworks that we use in our daily PHP development are not ready to work with lambdas because they were not designed for those kinds of architecture.\nImagine you are a PHP developer and you want to run serverless microservice using one of the most popular frameworks like Symfony or Laravel. Imagine you can develop your microservice locally and just deploy it as lambda functions. Is it possible with PHP\nprogramming language? Yes, it is. Even if AWS Lambda does not have native support for PHP. Since AWS lambda supports custom runtimes, you can create one for PHP and run your lambda functions written in that language. There are plenty of articles which describe the process of creating custom runtimes, but believe me it's time-consuming and not super easy.\nLet's take a look at the bref.sh project and try to run a simple Symfony app as lambda. I assume you have knowledge about Symfony framework, or you have an existing project and some knowledge about AWS platform. Before we start please make sure you have installed: First, we have to add Bref to our project with As you may know, AWS lambda filesystem is read-only excluding Let's take a closer look at template.yaml Thanks to the configuration stored in template.yaml, Bref will create CloudFormation with two most important resources which are lambda functions that will handle all the requests. One called symfony-website and the second symfony-console, both names are defined under OK. Let's check how it works and deploy.\nFirst, we need to create s3 bucket per our app with simple AWS CLI command Finnaly deploy our microservice: After we receive output \"Successfully created/updated stack - Lambda Functions: API Gateway: Just check your execution url in API Gateway: Other drawbacks: But still I think this is a great starting point and I believe it can solve two major problems of the \"serverless PHP\": To have visual alerting we watch dashboards powered by Grafana, Kibana and OpsGenie on big TV screens hanging on the walls where we sit. For OpsGenie we use this dashboard with custom modifications. First version of screens were powered by Linux laptops connected over HDMI cables.\nThat was not scalable, was expensive and required manual support to update Linux,\nrestart laptops, etc. Later DevOps team's boss Jönas Fleer found a nice project ResinOS. This project (was renamed to balena) allows to attach Raspberry Pi devices to a nice dashboard, automatically upload and run Docker image on them and manage all of that in an easy and pretty way. Balena provides customized operating system that runs supervisor\nwhose task is to get updates of your Docker images pushed to balena cloud,\npull them locally and restart container with new changes. You may ask, why Firefox and not Chromium? Unfortunately Chromium is too heavy for poor Raspberry Pi device, even for 3 B+ model that we use. Here's a Dockerfile that balena builds and deploys on devices: Of course when you have many screens it would be inefficient to duplicate app just to open different web URLs in a browser. For that purpose we installed DashKiosk and configured all devices to open the same web link that was a DashKiosk URL. In DashKiosk you manage connected devices to open proper dashboards URLs. Here's profiles.ini added in Dockerfile above: And here's docker entrypoint script. It's purpose is to Once application is ready it's not the end of configuration. Because latest web tools such as Grafana or Kibana require quite some hardware resources we need to tune balena OS settings to give container more GPU memory and include additional video drivers. For that update device configuration or entire fleet with following variables: That's it, took me a lot of time to make it work on Alpine Linux. Result looks like this:\n Here, at AUTO1 Group, we are growing. Not only in terms of company size, number of employees, revenues, etc, but also in terms of database size, amount of servers, lines of code in applications, amount of accessible pages. For supporting this growing codebase we are adopting new approaches and new technologies. Different innovations were already introduced, but today I want to tell You about switch of a paradigm from MVC to SPA, and how implementation of SPA approach evolved. We have our internal application, called Admin, which is heavily used by hundreds of colleagues in many countries. Historically, it was written as a monolith php application, based on Silex (and later Symfony) frameworks (some overview is present in this blog article). Admin application had architectural layers, which fell into a concept of Model-View-Controller (MVC). The View part, which is in essence the HTML code of all pages, was prepared on web-server using TWIG templates. Usually, one developer was working on both back-end and front-end implementations of web-page. That was called Downside of such approach became obvious when first HTML tables with complex internal structure were implemented. Lets take as an example a page that displays hundreds of cars, provides pagination, possibility to edit data, shows aggregated results, etc. Such page consists of static components (navigational menu, list of available locales, information about current user, html-template, that renders appearance of the page, etc) and dynamic components (information about cars). This kind of web-page is And this was exactly the case, that fitted well into SPA paradigm. SPA stands for Single-Page Applications - an architectural paradigm, that divides elements of the page into static and dynamic. Static elements of the page (such as page layout, navigational menu, user information) should be loaded into client's browser only once per user session. The page layout rarely changes. On the contrary, dynamic components of the page can be updated continuously. Regardless of the technologies involved for rendering a page in client's browser, main difference between SPA and non-SPA approaches is whether a browser First picture represents traditional request where full page contents are passed from web-server to a browser on each request. Second picture illustrates SPA approach: full page content is fetched only once, afterwords only dynamic content is downloaded and full page reload does not happen. Starting from a monolith application, where HTML was rendered on a server-side, we gradually moved to our current state, where a separate application serves SPA for us. But let's go step by step. Adopting of new front-end technologies, such as JavaScript library React, allowed us to start building first SPA pages. React code is executed in the web-browser. Front-end (React) developers became responsible for all the visual representation of a web-page. For organic integration with new technology we updated existing workflow of generating HTML of the page. Specifically, we introduced several new endpoints, that provided essential information for rendering HTML page layout. Those endpoints were: Front-end developers wrote React code, which could fetch data from aforementioned endpoints and render HTML5-compliant page layout. Now we needed a way to pass this React code from web-server to client's browser during initial page load! This was achieved by introducing of what we call SPA-controller: set of php application's code (written with Symfony framework), which returned very basic standard html template that contained latest version of the React code. Being executed in the browser, this code rendered HTML contents of specific SPA page. (if template was not cached, or TTL of the cache is over) do a request to CDN, that contains actual version of front-end assets The response contains lightweight HTML similar to this: As You can see, SPA index page contains none of the data that should be shown on a page. This data will come as a result of subsequent requests.\nStages of building SPA page are illustrated in the diagram (notice 3 phases): As a result, we implemented a Golang application, that did exactly the same, as it's predecessor-monolith, but outside of php environment. The current page lifecycle started to look like this: This is pretty good, huh? Of course, there are challenges in current approach also. We are investigating the best way to deal with user locales in SPA index, we learn how to do proper caching and authenticating user, but this is a topic of another story! When I first started working on React/Redux, the very first choice of library I used for handling asynchronous actions was Redux Thunk as its maintained by the authors of Redux, and is very popular as well. Consider the following example, where an action type is being created for incrementing a counter and a corresponding action creator for it. Using redux thunk, an async action creator can be created as follows Pretty simple till now. Redux Thunk is easy to understand, but the problem arises when the application scales, and as the need for dispatching condition-based actions grows with the application, it leads to unmaintainable code. For example, consider the below code In the above code, there are too many callbacks and the code does not look clean. Also, it would become complex if more conditions are added to it later on. Then, how to simplify this? For our customer-facing websites like wirkaufendeinauto and AutoHero, we are neither using Redux Thunk nor Redux Saga. Rather, we are using a very simple and lightweight npm module Redux Actions Generators developed by our very own Alexander Afonin, who works as a Senior Software Engineer at Auto1.\nIf this module is configured as a middleware in the React application, the above code can be simplified as shown below Now you are ready to tackle the growing complexity with ease. Also, the code is clean and readable. One question that might arise is that why not use async/await with thunk actions which could have solved the problem that is being discussed earlier. But, there are advantages of using generators instead of async/await thunks which are listed below - Redux Thunk returns promises, which are more difficult to test. Testing thunks often require complex mocking of the fetch API, Axios requests, or other functions. With generators, you do not need to mock functions wrapped with effects. This makes tests clean, readable and easier to write. The best part about the Redux Actions Generator module is that you can catch all errors at one place, without the need to write multiple try catch blocks in different actions ( you can if you want to, but not mandatory ) With redux-thunk, this is not possible and you have to write error handlers everywhere, which makes the code look less readable.\nTo use this common error catching functionality, you can configure it while creating the store as shown below - Once you do this, your actions can become clean and more readable. For example, the below code... can be replaced with... In this way, all the uncaught errors will be caught with the Let's get to some real life example on how we use this pattern in our code. Consider the Auto Kaufen page on our wirkaufendeinauto website where we show a list of ads, filters, header, footer and some other content. For this module, we have some actions like Similarly, we have other actions like As you can see, in the above code we are making multiple API calls, some are in parallel, and some are fired conditionally based on the results of the previous API calls. This way our redux actions look clean and are easy to debug and maintain. So this is how you can make use of the ES6 generators to simplify your React application code. Comments and suggestions are welcome. Always try to keep your transactions short and write your queries to deal with records in a deterministic order with respect to other transactions. Almost every non-trivial system needs some kind of task (job, event, whatever you call it) scheduling functionality at some point. Having more than 250 micro-services, ours was no exception.\nAt some point, it made sense to us to provide such functionality as a service for our developers. With Considering the subject of the post which is An event on the server side is simply a record in the database. A simplified view of The following is a bird's-eye view of what happens on the service: Thread #1 b. The service processes the events and marks them as processed (by filling the Thread #2 b. The service marks them back as unprocessed (by setting The very first MVP was a simple read-modify-write (anti-)pattern (with the help of Spring, JPA, and Hibernate). It is not hard to guess to what issue this implementation is vulnerable: Deadlocks. In the beginning it was not a big deal, because deadlock errors were not frequent and we could live with it. However, very soon, after getting more clients, the deadlock issue (among other problems, e.g. lost-updates) started hurting the quality of the service (e.g. firing events twice). For the first attempt, we refactored the code to get rid of read-modify-write antipattern: Obviously this is an improvement, though reduced the number of deadlocks, it didn't solve the issue completely.\nThe quickest solution to eliminate deadlocks was to change the isolation level to The better way to deal with deadlock is to analyze transactions and queries to prevent transaction interference by: By the first refactoring, we made transactions a bit shorter. But there's still some room for improvement. To apply the second best practice from the above we refactored the queries as follow (essentially ordering and acquiring a lock): However, the deadlocks were still appearing intermittently: Of course! we still had one transaction/query dealing with records in a non deterministic way: After this last change, we saw the game for deadlocks was over: \"Goodbye Deadlock!\" Over time we realized that our microservices built on top of Spring framework take tons of time on the integration test step. Every single PR runs all tests and this takes 10-20 minutes for some services. To decrease PR build time we decided to research why this happens.\nOne of the problems that this research highlighted is ramping up heavy spring context several times during execution of all module’s tests. As a proof of concept we have taken 3 modules and cleaned up their tests to achieve better integration test performance. The results are pretty much self-explanatory: Below I list all factors that produce many spring contexts and how we managed to overcome them. Our spring-based microservices actively use spring profiles and configuration classes.\nMany components from common libraries are hidden under profiles and configuration classes to not be active for every single module / environment without need. So, integration test authors define profiles and configuration classes that should be activated for their tests using these annotations: Here mongo-test-two-nodes activates special mode for mongo based services to operate with two separate mongo instances. As not all tests need this, the profile is turned off by default and only those tests that need this behaviour would explicitly turn it on. But this, unfortunately, leads to multiple spring contexts being created. The same happens with AwsTestConfiguration that deploys special test-component to mock AWS calls. The solution was to make all tests use the same set of active profiles and context configuration classes for every single integration test.\nThe drawback is that some components are deployed, even for the tests that do not need it. But deploying single component takes O(1) time comparing with the whole spring context. Another point to keep in mind is that order of profiles (configuration classes) matters.\n Two settings mentioned above result in 2 different spring contexts. One interesting feature in spring-boot-test is mocking / spying real spring beans.\nThis was mostly used when we don’t care what some method does in test but we need to check that it was called by parent components.\nUnfortunately this also makes test to create separate spring context.\nWhat we’ve done here is replacing Another option is to use the combination of However, the drawback of the last solution is that, parentComponent needs to have a setter for childComponent which is not good in terms of encapsulation. But if you don’t use autowiring on constructor level it is fine. Anyway, one can choose whatever fits the situation better. One of the services used this annotation to clear the caches between different tests.\nClearing a cache is really a problem (see section below), but this solution comes to separate spring context per test class or even test method (depending on annotation parameters). Context per method was actually the case for a few test classes. Instead in this modules we added methods to clear the caches programmatically without deploying the whole new context.\nOne can use java reflection API instead of adding special production-level methods, again whatever fits you better. After all this “multiple context” factors are gone one may face (and we did) a challenge to fix or re-organize some tests. Before above enhancements were done all tests were running in separate spring contexts, therefore used separate instances of every component and therefore those problems could be hidden. As an example some tests in mongo-based service were testing The rule that I would suggest here is that every test should clean up whatever it needs to be clean and not rely on fairness of other tests. That was a problem in our case. Some tests had So, now after we know all this, we created two tools (and may create more in future) that help to identify those problems. After every PR commit we run one additional step that analyzes test execution log.\nIf more than one spring context is created this leaves some traces and a notification is sent to the author of the commit via slack As an enhancement, the jenkins jobs could be modified to fail the build if the number of spring contexts is greater than a certain threshold. The threshold could start with a high number and can gradually be reduced as the team cleans up the existing tests by adopting the aforementioned practices/fixes We wrote an IntelliJ plugin to identify and solve the aforementioned problems. There will be a separate blog post about this plugin coming very soon. Each Friday at 17:00, a slackbot called Darth Vader appears in our #random chat, shares some statistics about our week in AUTO1 TECH and officially declares beer’o’clock. While the fridge is more or less filled and usable throughout the week, it is being refilled to the brim on Friday mornings by our beer enthusiast volunteers. Some weeks ago, something terrible was discovered during breakfast, with only seven hours to go until Feierabend. Quickly after, nihilists and dark tourists took over the scene and waved their doomsday scenarios at the crowd or tried to ply out “funny videos” for their individual amusement only. Considering this happened in an engineering kitchen, solution design and problem solving ingenuity kicked in quickly. These were the contending approaches: At this time the crowd agreed that this epic challenge could only be taken on by a single hero, as the prophecy predicted. And it was known as Beerxcalibur. “Only the one who removeth the beer frometh the fridge can be considered the true king/queen” More solutions came in: While a dozen experts discussed solutions and got all excited about possibilities, constraints, KPIs and SLAs, math and physics, laws of gravity, risks and opportunities, one lonely guy without any contextual knowledge and with no video camera attached to at least four parts of his body went to get coffee, saw the bottle, opened the door just a little bit, let the bottle slowly slide down to the bottom, picked it up, sorted it into the shelf it belonged to, took a picture with his smartphone and went back to business. The picture, once posted to the beery solution slack room, caused several people to walk through all the stages of grief. And of course, someone had to ask the question: How many engineers do you need to fix the beer fridge. The answer is: 15. Well, one. But 15 engineers helping, you know ... \"Helping\". The AUTO1 story started in 2012. In these six years company grew a lot. From one website\nwritten in PHP it evolved into a huge and complex system consisting of large number of components.\nThis article describes the evolution of AUTO1’s architecture from a monolithic approach to Microservices. The brand independent automotive company owns business units like auto1.com, wirkaufendeinauto.de and autohero.com. At the beginning we build several classic monolith applications that served different parts of business. The first phase was a great beginning of our journey to the world of microservices.\nWe got amazing results and most of our problems were solved: Unfortunately there is no silver bullet. The microservice approach approach has its drawbacks and we faced some of them. To resolve last of this problems we introduced the SPA approach.\nWe started using React. It allowed us to call microservices directly from the frontend with proxying requests through our monolith applications.\nTo be able to do so, we introduced simple proxy layers just for communicating with services.\nMore about this approach you can read in our blog article. As a result our architecture took it’s next look: For the last six years we passed a long way from old fashioned php websites to huge distributed net of microservices.\nHere are some numbers at the end of 2018: Microservices brought us lots of advantages, but along with advantages new challenges had come. Since day one at AUTO1, we strongly believe that the microservice architecture helps us run business at scale not only due to the technical benefits it brings but also, if not foremost by allowing many teams to work independently from each other. Today, we’re running more than 250 microservices in production and on average add one more to our platform every week. All of them are powered by the Spring Framework technology stack, which we value very much for its productivity, but at the same time consider it to be a bit heavy in terms of bootstrapping time. The typical startup time of our Spring applications varies between 40 and 80 seconds. Since we do around 30 deployments per day, our engineering teams spend roughly 30 minutes daily waiting for the deployments to be completed. As we value the time of our engineering teams, we wanted to find a way to decrease these numbers. While investigating the startup times of our applications, we have identified classpath component scanning as a potential place to look for improvements. Luckily for us, some of our tech savvy engineers suggested a not so prominent Spring 5.X feature to come at help here - the build time component candidate index.\nThe build time component index is an alternative for regular component scanning, which skips the classpath search operation in favor of using a compile time pre-generated index of component candidates during the spring application context building. However, it comes with a small print from the Spring Framework experts saying that this functionality should have mostly a visible impact on startup times for applications with large amount of beans and that are operating in environments where IO operations are expensive (e.g. remote file systems) or where JVM security managers are in use. Although, neither of this is our use case, we have decided to proceed with the feature evaluation and measurements, as here at AUTO1, we like our decisions to be data driven. The key part of the indexer feature is a pre-generated index file located in META-INF/spring.components one per JAR. Every index entry in this file is a fully qualified name of a candidate component as a key and comma separated stereotypes as value. So for example “X=Y, Z” can be read simply as register a candidate component X with following stereotypes Y, Z. Below is an example of a spring.components file: The first question that arises here is how to create such a file and keep it up-to-date. The Spring engineers thought about this problem as well and came up with an annotation processor tool called spring indexer that hooks in during the project build phase and generates the file. Making use of it is fairly simple, assuming you are using your favorite build automation tool, in our case maven and want our IDE to keep this file up to date, simply add the spring indexer annotation processor to your pom file: Once this is done, what happens during project build time is the spring context indexer (precisely speaking the CandidateComponentsIndexer class) while going through the project identifies annotated components and outputs them to META-INF/spring.components. The type of components it includes in the candidate list are: Classes annotated directly or indirectly with spring @Component annotation (which by itself is meta annotated with @Indexed) and other stereotypes for which @Component is a meta annotation such as @Repository, @Controller, @Service, @Configuration. Classes and interfaces annotated with annotations from javax package, including CDI annotations (@Named, @ManagedBean), JPA annotations (@Entity, @EntityListeners) or even Servlet annotations (@WebListener) Any custom classes and interfaces annotated with @Indexed annotation. It should be noted that applying @Indexed on a class/interface where no other standard stereotype (as from points 1 and 2) is present will not make by itself the bean managed, thus not making them eligible for being autowired, but merely provide a hint that it represents a stereotype to be considered in the index. Example of the @Indexed annotation usage: Or as meta annotated as below: In order to make them available in the application context, thus autowirable, we could for example include them in the component scan filters as shown below: There is one big pitfall that comes with the feature though, that is by default the index is automatically enabled when a META-INF/spring.components file is found on the classpath during application start-up and this in turn, disables completely the regular classpath component scanning. In other words, both the index and classpath scanning cannot be used at the same time. The consequence of this is that if an index is available for some JARs, but is incomplete due to including other JARs that come with potential bean candidates but don’t have the index file generated, those beans will not be discovered, thus application startup might spectacularly fail with a well known error: “X required a bean of type 'Y’ that could not be found”.\nFortunately, Spring engineers, as thoughtful as they are, provided a feature flag spring.index.ignore that can be passed either as a system property or set in the spring.properties file at the root of the classpath allowing us to disable the usage of the candidate component index altogether. Like mentioned before, the aim of our investigation was to understand if the Spring indexer feature is of any help to us in regards to boosting the application start-up times. In order to make the test fairly reliable and control as many confounding factors as possible, we have decided to run the test firstly, using a “dummy” Spring 5 application (in two flavors - with 120 and 5000 beans) using all the features a production microservice would use, on a bare metal machine i.e. a MacBook Pro late 2017 i7. Secondly, running a real life production service in a less controlled but close to production environment i.e. on a r4.8xlarge EC2 instance with other services running concurrently. The applications themselves were using Spring 5.0.10.RELEASE with Spring Cloud Finchley.SR2 release train.\nAfter making a couple of hundred measurements of application bootstrapping times with candidate component index enabled / disabled and verifying in the logs that Spring was indeed using (or not) the component candidate list, we ended up with following averaged results: As the above results have consistently shown, there was no improvement or even insignificant worsening (which will be a subject of our analysis in a separate article) of the bootstrapping time of the Spring applications, regardless of the number of candidate beans present, when using the pre-generated candidate component index. This was expected, as explained in the opening section, the conditions for which this feature was designed (slow or expensive IO operations) are simply not met in our case. One more important factor to consider here, is that a great deal of the beans in our Spring applications, is created using bean definition factory methods, which are not subject to the component classpath scanning, thus are not impacted by this feature. All in all, we deemed this feature not to be useful for our purposes and moved on to the next item on our list. (*) If your career as a QA - like the author’s - started without a computer science degree, you will hopefully love\nto hear and learn about the different theories and notations of runtime analysis.\nCheck wikipedia’s page about O-notation here Speed… Did you ever think how you can make your tests faster and more reliable? If you're anything like me or my colleagues at AUTO1,\nyou face this challenge all the time. Here at AUTO1 we are constantly looking for new ways to improve our tools and\nto make testing fast and efficient. In this article, I’m going to take test suite that takes ~70 min and reduce the execution time to 1 min!!!\nInteresting, right? :) Let’s imagine we are testing some Web Shop Scenario. We need to test that users can do some actions with items they purchased\n(submit feedback, return, submit claim, etc.). And imagine that in this case we have exactly 20 scenarios that go like this: Let’s look at the timeline of our test We significantly increased the speed. From ~73 to ~8 minutes. This already looks like a game changer! But can we do more? Spoiler: Yes we can! If we check our test scenario again, we can clearly see that the actual validation of our test scenario happens only\nafter the third minute! And everything we have up until then it’s just creating preconditions.\nWhat we actually want to test is: “User can do certain action with item purchased on our platform”.\nWe should not care HOW we got this item, we should have another test, testing that user should be able to purchase.\nThis just creates dependency for us that makes our tests: P.S. While writing this article I tried to re-create all these conditions and I didn’t have any successful build.\nAnd it was always something related to other dependencies. Solution? There are many possible answers, the one I chose for this example is to move all logic with test data preparation to the API layer.\nLet’s try this and review our timeline again Amazing, this is how our test looks now! two minutes just gone. And this is how our suite looks now comparing to other runs.\n Having test preparation using API is a great step here, but does it scale from 20 tests to 1000 without having side-effects?\nAt the end, your test data preparation might look like this: Assuming you have around 40 calls with a reasonable fan-out for every scenario and you usually run tests in multiple threads,\nyou might involuntary start a load test there. Aaaaand it’s time to look for a solution again! What if we have our test data waiting for us somewhere? Wouldn’t be amazing?!\nOur solution for this problem was to create a Web Service that would keep the data in a queue for us.\nIt provides couple of huge benefits for us: Let’s look at our timeline again. This is how our test scenario changed: We barely can see it on our timeline :) Stay tuned! We will open-source it soon! In AUTO1 Group, every day we are looking for smarter ways to do software development and testing.\nAnd the main point in this article is that you should never stop innovating and looking for new solutions to existing problems.\nSomething that works for us, is not necessarily going to work for you,\nbut maybe you will ask yourself: “Where is our next level? Can we do something better?” Using Docker Compose we are able to quickly set up complex environments consisting of multiple containers, which allows to speed up software evaluation process. Things evolving quickly, tools come and go, last year cutting edge is often legacy now. Dozens of new tools emerges every month\nand approach for being up to date also needs to evolve. Containers on production with all its benefits and challenges is a common thing nowadays, but do not forget that containers could also help us in other use cases. One of them is to improve technologies evaluation process. Many moons ago when new fancy tool was released and looks so promising to give it a try options were limited - build it from sources, use installer or package manager. Giving a try for new promising tool was quite time consuming. Steps required to download, install, configure system and dependencies were usually not most exciting part of evaluating new message broker or database. Now containers or PaaS solutions allow us to shift focus directly from setup to features that particular products offers. Using Docker one terminal command is enough to download, run and evaluate interesting applications. Above example command will pull latest version of PostgreSQL image from Docker Hub and run it locally. This is a quick and straightforward way of running hundreds of interesting applications. Container will run in sandbox environment, so we do not need to think about operation system settings and other specific configuration that may be required with standard installation. Although it is a simpler way than using installers or building from sources, then comparing to common package managers where also one command is usually enough, user experience looks quite similar. The real benefits of using Docker in local environment we can observe if we would like to build some complex environment that contains multiple applications linked together as usually things are more complicated than just running a database or monitoring tool separately. Challenge often starts when we would like to evaluate a monitoring tool that needs a datasource and entire environment should be relatively easily to set up and manage. That is where As stated in official documentation Compose is a tool that allow us to define and run multi-container applications. Using it in most common scenarios comes down to creating YAML file with required images and configurations. Having Docker will build or download all necessary images, set up necessary configurations, networks and run defined containers for us. In below example we would like to see in action Grafana - open source monitoring tool with PostgreSQL as datasource. First thing w need to do is to create empty Above we have defined one service called Having our datasource configured we can extend our Compose file with second service - Grafana. Above we have defined second service called At this point we have configured our environment, so by running While I thought that the visionary part and the progress that has been made so far was pretty cool, I had a million of technical questions, which unfortunately weren't being read from the web summit in-app possibility to submit questions to the talk hosts. The DNA that CATALOG is using is artificial, nevertheless my main burning question was \"Would real life viruses be a threat to those DNA-'Disks'?\" Are hackers of the future writing their own plasmid ring vectors with artificial virus DNA to destroy compromising data? That sounds like a cyberpunk short story waiting to be written. Similarly visionary questions are being asked by companies and founders who want to commercially explore space. It never occured to me, that companies like SpaceX or the cubesat projects where just the tip of the iceberg on commercial space usage. What I learned is that there are companies like Vector Launch, that sell launches of small scale rockets that can be chartered for surprisingly low prices (50kg payload for 1.5M$, which is around 35 cubesats!). Should I become rich one day and stop caring about money, I will launch a crate of Scottish whisky into space as a measure of refining, in hope that there is also a company that salvages that stuff from space. Of course fun and leisure has to be there as well, so there was also Virgin Galactic, a company that is tightly working on making space tourism a reality. Their presentation was pretty convincing. Maybe I'll save the money for the whisky and launch myself - given I'd be rich. A ticket to space currently costs 250.000$. An idea that totally baffled me was the what3words project which I find personally absolutely amazing and which in my view is just one billion-people integration away from a breakthrough. These people divided earth into a set of 3x3m squares and addressed each of them with three simple words. My desk at work is //abfinden.hübsch.ansehen - there you go! Pay me a visit. They recently integrated into the mercedes benz navigation systems and if you imagine one of the really heavy traffic websites and apps out there using what3words as a location service instead of GPS, just over night large portions of mankind would start using it.\nAfter coming home from the web summit, I found out what all kinds of w3w I have in my personal environment and I now have a w3w for my desk and my wife has one for her favourite spot on the couch. The fridge w3w is popular, the one for the dishwasher isn't. You gotta love the future! I truly enjoyed my time at the Altice arena, including the night-summits and the cheap cab rides. If time allows, I will definitely try to catch the 2019 web summit. While I sometimes missed the technical deep dives on topics, it was deeply satisfying for the curios me to explore, watch and meet the hundreds and thousands of booths, speakers and enthusiasts from all over the world. I’m so going to build my own cocktail robot! \"We need to re-write EVERYTHING!\"... It's understandable that a huge technical debt is created during the first period of building a company app, a lot of business needs, a lot of deadlines and smaller teams. And for webapps where a proper frontend solution was not considered, most of this technical debt falls on frontend and by time it becomes harder and harder to refactor till it reaches a point where the right way to refactor it is to re-write it. But that's not an option sometimes, isn't it? As an alternative, This article will introduce two approaches to integrate lightweight react apps (we will call them widgets) that can be embedded in any project. React widgets implemented directly in your codebase. Create a similar folder structure to this: For example Now running Now you can use a widget in one of your views like this In this approach we will keep all frontend in a separate repository and have an npm script to generate an html and bundle js file for each widget and upload them to a CMS. But first, A new database model need to be introduced that will hold the widget data (name and html) and there should be a way to upload assets to a content server. For example you may expose two web service APIs similar to this: The folder structure will be similar to the previous approach but this time in a separate repository: NOTE: I encapsulated the logic to upload the bundle file and save widget entry in In And now to deploy the login widget: Going back to the backend repository, You can render a specific widget using the html value saved in DB, for example For The story about how one hackathon project sped up the development of our back office system by 2 days per feature. The story about how we enabled our teams to develop in the newest technology-agnostic stack by extending the architecture of legacy monolith application. The story about how we enabled our teams to extend a monolith back office without touching it. In AUTO1, as in the majority of other companies, we also have a back-office software (aka “Admin”) where our employers can manage our business operations. We’ve got a lot of business processes, and therefore a lot of business logic put into the code. A couple of facts about Admin: From these facts, you can see, that what the thing we’re talking about - it's a typical monolith. An authentication between requests was handled with a very common approach for PHP applications - PHP Sessions. With this approach the session identifier for the current user is put in a cookie, and on the server side, the session data is somehow stored as a document. Then we’ve got our first microservices. Some business logic started to live its own life. And of course, our Admin should have had some control over it. So our Admin started to interact with our microservices, and initially it was treated more or less as a third-party API. Our microservices are located behind an API gateway (we call it GateKeeper), every authenticated request should be signed with an OAuth token to get through it. Also, every request should be signed by the client application with an API key. Admin has become one of the external clients of microservices API. It had its own API key, every user was obtaining an OAuth token, which is kept in a session document, as a part of a session data. Months were passing. In general, an architecture stayed the same, apart from the fact the number of our microservices increased. And increased a lot. More and more business logic was covered by services while less and less logic was implemented in Admin. Then at some point were realized, that what happens in Admin - it’s just a proxying of requests to a gateway. The usual code for a new feature was more or less - send a request to the Gateway, get a request, send this request to a front-end client. Not the most exciting engineering work ever. Moreover, we’ve got another complication - some teams without any PHP engineer had to add some new features to back-office. For this, one needed to create a proxying code, and for this one would need some support from another team. So we’ve started to think of possibilities to send requests to the Gateway directly from Frontend, in order to avoid a necessity to make any implementation in Admin every time we do a new feature.\nAnd challenges were: Then unexpectedly a group of our engineers (Mikhail, Oleg, Alexey and Andy - thank you, guys!) came out with a prototype of Vigolante, as a Hackathons project. Vigolante - a mix of GO and VIGILANTE (sp.), which means a guard. The idea behind it can fit in one sentence: it’s an authentication layer, which uses the same cookie as Admin, gets a session identifier from a frontend request, connects to the sessions storage (Memcached), fetches a session data, gets a OAuth token from the data and, enriches the initial request with the token and then forwards it to a Gateway. Quite a simple flow, but once in place helped us to save a lot of development time in order to deliver new features to Admin. From now on, we don’t need to make any changes in Admin, when we need to send a request to a microservice from Frontend. We have chosen Golang as an underlining technology to build the project. The reasons are: Basically, the project was developed by the efforts of 4 engineers in a bit more than one day. We are proud to say, that this time was not just for coding but also for taking care of bringing the software to production; things as: There were few things yet to be resolved (such an implementation of mechanisms of refreshing of OAuth tokens, also there were some problems with encryption of PHP sessions from Memcached), but those were successfully resolved; one month later we released the first feature relying on Vigolante to production. Feeling proud and fulfilled, we could now focus our attention on the our engineering challenges. Besides using aggregate metrics gathered from machines and cloud resources, we extensively make use of our log data for monitoring. For this purpose we introduced Yelp’s ElastAlert to our monitoring setup some time ago. It queries our ELK cluster and sends notifications to teams in charge. Sometimes elastalert returns unexpected results, sometimes it does not alert although one would expect an alert. In this article I want to explain some of the common questions we have asked ourselves while rolling out elastalert: Elastalert runs all rules in a loop on a fixed schedule. In the most use cases this means it loops over all rules once per minute. As events are usually not indexed in real time it adds some buffer_time (per default 15min) to the timeframe in which it searches for events. On each run elastalert filters the returned events based on the given timeframe and stores the result in memory . Also, if number of search result > threshold(num_hits) configured in the rule, it will directly alert. If following runs return new events it has not stored so far (based on document id) elastalert adds these and once it reaches the threshold alerts. It works like a sliding window over events. This behavior allows for late indexed events due to pipeline congestion which especially in error situations might happen due to increased event numbers. Enabling use_count_query for frequency rules might sound like a good option to lower the load on the system(count queries are usually faster than real searches as no data has to be transported). However this deactivates the complete event filtering and storage on elastalert side. Still the rule is executed each run period(once per minute) and the counted query hits are stored. As the count query returns no timestamp elastalert uses the execution time to determine if a a rule has reached its threshold. If events are not indexed in realtime this easily leads to lost events to alert on. To compensate this set the query_delay parameter of a rule to at least one minute. Below you find a simplified view of how normal rules and rules with use_count_query are handled and why in most cases its better to not use_count_query. Each time a rule is reloaded or Elastalert is restarted the complete in-memory buffer gets discarded. This resets threshold counter and the sliding window starts new. For rules which have timeframe < buffer_time (so on default setting <15min) all events leading to a potential alert will be reread into elastalert. If the timeframe is > than 15min this is no longer the case. For this the property scan_entire_timeframe is available. However for frequency rules with use_count_query elastalert would have to set the timestamp of all events the rule matches to the startup or reload time of the rule. So they will be considered a whole timeframe long for alerting which could lead to a lot of false alerts due to a different assumption of the time window elastalert considered then manually checking the events. If your rule only reached its threshold after running multiple queries in the timeframe you can add the old events by using the attach_related property for your rule. If that creates unreadable or to long alerts you can also just send interesting fields by using include: [„x“,“y“,“z“]. When managing lots of alerts from different sources it might be tricky to find the sweet spot between a very specific index pattern (like one just for this type of events) and a broader one which would also include other events. In our experience its better to just specify a very broad index pattern in the alert rule and let elasticsearch do the job figuring out the indices to search through. This allows administative changes later like introducing new rollup patterns or splitting/merging indices. To minimize startup & reload issues, either be ok if you loose some matches on elastalert restarts. Or try to not using a timeframe higher than buffer_time (15min). Use elastalert-test-rule to test your rule before deploying it and ensure special chars in strings are properly encoded etc. Otherwise small errors in the query might easily be missed and then the rule never alerts. And use query_key to minimize the number of queries to be sent to elasticsearch. F.i. by using it to monitor if a certain service has returned more than threshold of any http error status code with a common threshold. Few weeks ago AUTO1 started an exciting journey, looking for talented people in Silesia in Poland. Today we happily announce that on 02.07.2017 we had the pleasure to start our first Tech center in Katowice, Grabowa 2\nstreet. We are proud of having found talented, motivated and passionate engineers! It's not all about engineering! That's quite a statement from a software engineer, but I do believe that's a true fact\nin our life as professionals: sometimes you really need time to hack things together. We just had our fourth Hackathon\nover the last weekend and though I could not contribute to it actively with a project of mine it was a tremendous amount\nof fun! Unfortunately, the BBQ we were hoping to have got cancelled due to bad weather conditions - thunderstorms showering\nBerlin repeatedly during the two days and the first night of the event. We had a fallback plan of course, so we simply\nresorted to it, with a massive pizza delivery ordered for dinner. If I were superstitious, I would likely expect at\nleast some bad luck from a date like Friday 13th... The start of the event is marked by a very informal pitch of project ideas from all the teams, that happened in one of\nour lovely kitchens in our Berlin Headquarters. I cannot make it to the meeting, but thanks to Lukas this moment is\ndocumented in pictures I can enjoy later during the day. After the presentation the vibe of the Hackathon starts to percolate through the whole department, with groups of people\ndiscussing the nitty-gritty details of their plans: people start grouping around whiteboards, chatting in a kitchen over\na coffee or a cup of tea. Countless hours of hacking follow, seldom interrupted by breaks and further cross-team chats,\nuntil those three delivery guys make it to Engineering, carrying three huge piles of hot pizzas. Most participants leave\nbefore eleven, just a couple of hours - and of course a few drinks - after dinner. Day two doesn't start with the best premises either: one of the entrance doors got broken, forcing everybody to find\ntheir way in through a secondary door most people were unaware of until this very moment. When they finally make it,\nthey find a nice breakfast provisioned by this guy, earlier in the morning: bread, butter, berries jam, mortadella,\nprosciutto and salami... that's what you get when you send the Italian guy out for the groceries... Back to work, one more day of hacking. Hours fly by and in the afternoon everybody starts to look tired, with people\nleaving a few at a time when they either reach their goals or simply drop the towel. The last man standing leaves the\noffice at 9 pm, when I finally lock the door behind me and leave as well. We won't meet at the office on Sunday, though\nfurther remote work is allowed, for the overly-committed. The weekend is over, we are back to work. As the day is getting closer to its end, I see a few colleagues around the\noffice working on their projects presentations - the election of the winning teams is about to happen. We gather again\nin a meeting-room and groups take turns at the beamer, showing everybody else the best of their hacks. We all enjoy\nseeing our colleagues presenting implementations ranging from innovative to crazy and from business-focused to\nfun-making: cloud-based real-time analytics of business events, The-Matrix-style error logs visualization,\n\"The Mini Kevin 500\" - ok, this one would take some time to explain, in a nutshell, it's a hardware monitoring tool\nnamed after our CTO - and, to our greatest delight, somebody built a 3d printer for our office! We do our usual voting by popular acclaim and the winners finally get some brand new Nerf guns, the well-deserved prize\nfor their efforts! So once again, it’s not all about engineering, but also making crazy ideas become real and most importantly have fun\ntogether. And that’s why AUTO1 is such a great place to work! Transaction isolation level defines the degree to which one transaction must be isolated from resource or data modifications made by other transactions. Let's have a look at phenomena which can occur during the execution of concurrent transactions: Dirty read - A transaction may read data written but not yet committed by other transactions.\n example: Transaction T1 modifies a row. Transaction T2 then reads that row before T1 performs a COMMIT. If T1 then performs a ROLLBACK, T2 have read a row that was never committed and that may be considered to have never existed. Nonrepeatable read - A transaction re-reads data it had previously read and finds that data has been modified by another transaction (that committed since the initial read).\n example: Transaction T1 reads a row. Transaction T2 then modifies or deletes that row and performs a COMMIT. If T1 then attempts to reread the row, it may receive the modified value or discover that the row has been deleted. Phantom read - A transaction re-executes a query returning a set of rows that satisfy a search condition and finds that the set of rows satisfying the condition has changed due to another recently-committed transaction.\n example: Transaction T1 reads the set of rows N that satisfy some search condition. Transaction T2 then executes SQL-statements that generate one or more rows that satisfy the search condition used by transaction T1. If transaction T1 then repeats the initial read with the same search condition, it obtains a different set of rows. Isolation is the I in the acronym ACID. In ACID, atomicity and durability are strict requirements. Whereas consistency and isolation are more of configuration (kind of). Besides, they are closely related, so much so that the SQL standard defines four levels of transaction isolation based on the consistency they provide (from least to the most consistency): Different isolation levels are defined based on whether they allow (or prevent) the above phenomena (as described by the SQL standard). Note that transaction isolation does not affect the changes made by the same transaction. It means a transaction always sees all the changes made by itself. Inside Java services, the transaction isolation level can be controlled by Spring It is worth to mention that default isolation levels for each system might be different. For example, the default isolation level for MySQL 5.7 (InnoDB) and PostgreSQL 9.5 is respectively, REPEATABLE READ and READ COMMITTED. To check the default isolation level, query: Everyone would agree that security is very important and with the increase in cyber attacks in the recent past, organizations are investing\nheavily on application security. In this post lets talk about securing a web application built in NodeJS. NOTE: The security concepts discussed here are language or framework agnostic. However, here we will see how these practices are implemented\nin NodeJS web applications. It is always a good practice to send your data over HTTPS rather than HTTP and it is imperative if your app transmits sensitive data.\nEncrypting data transmitted between the client and server helps mitigate several attacks like man-in-the-middle(MITM) attack, packet sniffing,\neavesdropping etc. Let’s see how to set up TLS/SSL in Express 4.x: Lets first generate a self-signed certificate: This generates a self-signed certificate valid for 365 days. NOTE: The self-signed certificate is not ideal for production. For production, you should get a certificate from a Certificate Authority(CA). Next, enable HTTPS on Express. Additionally, redirect all HTTP traffic to HTTPS: i) Strict-Transport-Security: The HTTP Strict Transport Security(HSTS) if set in the response header, tells the browser that it should only communicate using HTTPS instead of HTTP while communicating with the specified domain. Syntax: Here, max-age is the time(in secs) that the browser should remember that this site is only to be accessed using HTTPS. Example: from facebook.com: ii) X-Frame-Options: This HTTP response header can be used to indicate whether or not a browser should be allowed to render a page in a Syntax: iii) X-XSS-Protection: This HTTP response header enables the built-in XSS filter in modern browsers. Example: iv) X-Content-Type-Options: This response HTTP header is a marker used by the server to indicate that the MIME types advertised in the Content-Type headers should not be changed and be followed. This prevents MIME type sniffing attacks. Syntax: v) Content-Security-Policy: Prevents a range of injection attacks including Cross Site Scripting(XSS) attack. Syntax: For a detailed explanation of CSP, go through this link. To set these headers in NodeJS, use the helmet npm package: This sets all the necessary headers in response. To set the headers individually: For an exhaustive list of security headers that should be set in a web application, check out the OWASP Secure Headers Project. NOTE: In some web servers, the security headers can be set in the server configuration file itself. For example, in nginx server, we can set the above headers in nginx.conf as shown below: Cross site request forgery (CSRF), also known as XSRF, Sea Surf or Session Riding, is an attack vector that tricks a web browser into executing an unwanted action in an application to which a user is logged in. CSRF attacks specially targets state-changing requests and can force the victim to transfer funds, change email/password and so on. CSRFs are typically conducted using social engineering, such as an email or link that tricks the victim into sending a request to a server on behalf of the attacker. The server has no way to distinguish a forged request from a genuine one. In NodeJS, to prevent CSRF attack, we usually use the csurf express middleware: In the view use the CSRF token passed: Cross-Site Scripting (XSS) attacks are a type of injection, in which malicious scripts are injected into otherwise benign and trusted web sites.\nAn attacker can use XSS to send a malicious script to an unsuspecting user. The end user’s browser has no way to know that the script should not\nbe trusted, and will execute the script. Because it thinks the script came from a trusted source, the malicious script can access any cookies, session tokens, or other sensitive information retained by the browser and used with that site. These scripts can even rewrite the content of the HTML page. The thumb rule to prevent this category of attack is to always validate and sanitize user data before processing or storing in database. Never trust data coming from user.\nValidation must be done on the server-side as client-side validation can be easily bypassed using tools such as Burp Suite, TamperData etc. A common approach to validate and sanitize user data is to use a library like validator.js. Example: To validate an email This library provides a number of validators and sanitizers to filter user inputs. Other useful libraries include DOMPurify and xss-filters. Here is an example to sanitize user input using xss-filters: Passing unvalidated user input directly to a SQL statement is vulnerable to SQL injection attack. Conside the following example: Now suppose the user enters the following in the username field: The above SQL statement becomes: This effectively nullifies the need of a password and returns all the users in the database. This attack can be completely prevented by using parametrized or prepared statement. If you’re using an ORM to access the database (Mongoose, Sequelize etc), the ORM will normally take care of SQL injection by using prepared statements under the hood. XSS vulnerability in an application can be used to steal browser cookies. To prevent cookie stealing we can set the httpOnly flag of the cookie.\nAdditionaly, we can tell the browser to send cookies only over HTTPS using the secure flag. secure : this attribute tells the browser to only send the cookie if the request is being sent over HTTPS. HttpOnly : this attribute is used to help prevent attacks such as cross-site scripting, since it does not allow the cookie to be accessed via JavaScript. Example: To prevent our site from overwhelming with a large number of requests, we need to put some kind of rate limiting to our API. We can use the ratelimiter npm package to implement rate limiting. If you are using Express, the express-rate-limit middleware can be used as shown below: Any error in the application should be handled gracefully by showing a custom error page to the user instead of showing stack trace in the error page\nthereby leaking sensitive infrastructure information like server info. i) The Node Security Project\nTo check the various npm modules for known vulnerabilities, the Node Security Project provides the nsp tool to check for vulnerabilities: ii) Synk\nSynk checks the application against Snyk’s open source vulnerability database for any known vulnerabilities in our dependencies. iii) nmap\nNmap (“Network Mapper”) is a free and open source utility for network exploration or security auditing. iv) sqlmap\nsqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. v) Burp Suite\nThe must-have tool for application penetration testing. It includes an automated scanner to detect most common vulnerabilities in a web application. OWASP Top 10 Most Critical Web Application Security Risks This is our first blog post. There must be millions of blogs that started exactly like this. And there must be millions of engineers, Tech enthusiasts, computer scientists, script kids, “soon to be hackers”, and curious people out there who typed these virtually legendary words into their development environments, just to see them printed on screen seconds later. We’re smilingly shaking hands with tradition. While we have been more on the introverted side of things for the past five years, focussing on building a platform that is the cornerstone of a multi-billion Euro business, we're proud of the culture and stack we have been building and want to use this channel to open ourselves up from now on. We And pass the blinking cursor to our editorial team. Yours truly, The AUTO1 Group Tech Department. \n@Test\nvoid givenHungryThenCook10BreadsWithMyWife() throws InterruptedException {\n var basket = new ConcurrentLinkedQueue<>();\n var knife = new Knife();\n var startGate = new CountDownLatch(1);\n var endGate = new CountDownLatch(10);\n\n Function<String, Runnable> preparation = topping -> () -> {\n try {\n startGate.await();\n } catch (InterruptedException e) {\n throw new RuntimeException(e);\n }\n synchronized (knife) {\n var bread = new Bread();\n knife.setTopping(topping).applyToppingTo(bread);\n basket.add(bread);\n endGate.countDown();\n }\n };\n\n // (...) omitted for brevity\n}Thread blocking
\nNonblocking Algorithms
\n\nclass Counter {\n private Integer count = 0;\n\n public void incrementCount() {\n this.count = count + 1;\n }\n\n public int getCount() {\n return count;\n }\n}\n\nclass ThreadSafeCounter {\n private AtomicInteger count = new AtomicInteger();\n\n public void incrementCount() {\n this.count.incrementAndGet();\n }\n\n public int getCount() {\n return count.get();\n }\n}\nprivate int value;\n\npublic final int incrementAndGet(int expectedValue) {\n int oldValue;\n do {\n oldValue = value;\n } while (!compareAndSetInt(oldValue, oldValue + 1));\n return oldValue;\n}\npublic class Knife {\n private final AtomicReference<String> topping = new AtomicReference<>();\n\n public Knife setTopping(String topping) {\n while (!this.topping.compareAndSet(null, topping)) {}\n return this;\n }\n\n public Knife applyToppingTo(Bread bread) {\n String currentTopping;\n if ((currentTopping = topping.getAndUpdate(s -> null)) != null)\n bread.setTopping(currentTopping);\n return this;\n }\n}Immutability
\n\npublic record Knife (String topping) {\n\n public Knife setTopping(String topping) {\n return new Knife(topping);\n }\n public Knife applyToppingTo(Bread bread) {\n if (topping != null)\n bread.setTopping(topping);\n return new Knife(null);\n }\n}\n@Test\nvoid givenReferenceToRecordThenItShouldBeImmutable() {\n record Parent(String name, List<String> childrenNames){};\n var childrenNames = new ArrayList<>(List.of(\"Haru\"));\n var parent = new Parent(\"Bruno\", childrenNames);\n\n var parentBeforeNewChild = parent.toString();\n childrenNames.add(\"Ayumi\");\n\n assertThat(parent).hasToString(parentBeforeNewChild);\n}\njava.lang.AssertionError: \nExpecting actual's toString() to return:\n <"Parent[name=Bruno, childrenNames=[Haru]]">\nbut was:\n <Parent[name=Bruno, childrenNames=[Haru, Ayumi]]>\nrecord Parent(String name, List<String> childrenNames){\n public Parent(String name, List<String> childrenNames) {\n this.name = name;\n this.childrenNames = new ArrayList<>(childrenNames);\n }\n};\n@Test\nvoid givenReferenceToRecordThenItShouldBeImmutable() {\n record Parent(String name, List<String> childrenNames){\n public Parent(String name, List<String> childrenNames) {\n this.name = name;\n this.childrenNames = new ArrayList<>(childrenNames);\n }\n }\n var childrenNames = new ArrayList<>(List.of(\"Haru\"));\n var parent = new Parent(\"Bruno\", childrenNames);\n\n var parentBeforeNewChild = parent.toString();\n parent.childrenNames().add(\"Ayumi\");\n\n assertThat(parent).hasToString(parentBeforeNewChild);\n}\njava.lang.AssertionError: \nExpecting actual's toString() to return:\n <"Parent[name=Bruno, childrenNames=[Haru]]">\nbut was:\n <Parent[name=Bruno, childrenNames=[Haru, Ayumi]]>\nrecord Parent(String name, List<String> childrenNames){\n public Parent(String name, List<String> childrenNames) {\n this.name = name;\n this.childrenNames = List.copyOf(childrenNames);\n }\n}Summary so far
\n\n
\nConclusion
\nAppendix
\n","fields":{"slug":"/a-wild-thread-appeared-multithreaded-testing-and-thread-safety/","tags":["Thread-safety","testing","nonblocking_algorithms","immutability"]}}},{"node":{"id":"4762bbde-b461-512b-93ae-22335b823a90","frontmatter":{"category":"Architecture","title":"Harmonizing Application Cockpit data using DB views","date":"2023-02-16","summary":"Our journey building the AUTO1 service inventory search using DB views","thumbnail":null,"authorName":"Mariusz Sondecki","authorDescription":"Mariusz is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/harmonizing-application-cockpit-data-using-db-views/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAWABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAMEBQH/xAAXAQEBAQEAAAAAAAAAAAAAAAACAwAB/9oADAMBAAIQAxAAAAGv2Svu3GUctOAU5wh//8QAGxAAAQUBAQAAAAAAAAAAAAAAAQACERIiEyH/2gAIAQEAAQUCJkDB7KBW+bEJwkNZ6v/EABgRAQADAQAAAAAAAAAAAAAAAAEAAhEQ/9oACAEDAQE/AQMmMLPP/8QAGBEBAAMBAAAAAAAAAAAAAAAAAQAQESH/2gAIAQIBAT8BV2cmFf/EAB4QAAEEAQUAAAAAAAAAAAAAAAABAhEhQRAxUWFx/9oACAEBAAY/AqwSjpMj+oKvRU5PDZD/xAAdEAACAgIDAQAAAAAAAAAAAAABEQAhQWExcZFR/9oACAEBAAE/IWdbBmVNkJTg+IpKCEBHylkKGgktmegIYKT9dwgOydif/9oADAMBAAIAAwAAABAA18H/xAAXEQEBAQEAAAAAAAAAAAAAAAABEQAQ/9oACAEDAQE/EES8YEurv//EABgRAAIDAAAAAAAAAAAAAAAAAAEQESEx/9oACAECAQE/EDJJlS//xAAdEAEAAgICAwAAAAAAAAAAAAABABEhQVFhMZGx/9oACAEBAAE/ECQu0QVxCvgawa6iFVCFbUJDi/sW7HoRRveCLN+4F4gKruoOH4KtotWHILP/2Q==","width":50,"height":50,"src":"/static/41d7e93878c0ba2eb12594d317c2b361/d2d31/avatar.jpg","srcSet":"/static/41d7e93878c0ba2eb12594d317c2b361/d2d31/avatar.jpg 1x,\n/static/41d7e93878c0ba2eb12594d317c2b361/0b804/avatar.jpg 1.5x,\n/static/41d7e93878c0ba2eb12594d317c2b361/753c3/avatar.jpg 2x,\n/static/41d7e93878c0ba2eb12594d317c2b361/31ca8/avatar.jpg 3x"}}},"headerImage":null},"html":"@Test\nvoid testTheCake() {\n record Cake(String name, boolean hasFruit, int weightInKg){};\n\n var chocolateCake = new Cake(\"Chocolate\", false, 5);\n var strawberryCake = new Cake(\"Strawberry\", true, 3);\n\n var bakery = List.of(chocolateCake, strawberryCake);\n\n var heavyCakes = new Condition<Cake>(cake -> cake.weightInKg >= 3, \"heavy cake\");\n var bigChocolate = new Condition<Cake>(cake ->\n cake.name.contains(\"Chocolate\") &&\n cake.weightInKg >= 5\n , \"big chocolate\");\n\n assertThat(bakery)\n .haveExactly(2, heavyCakes)\n .haveExactly(1, bigChocolate);\n}The challenge
\norg.springframework.boot:spring-boot-starter dependency in version 2.3.0 onwards:The dilemma
\nWhy not reuse what is already available
\nDynamically built queries
\nDatabase views
\ndocument_view, would be the aggregation of multiple sub-views. Each sub-view would only know about its data structures, hence we would achieve the desired single responsibility and encapsulation traits, making the solution more extendible and more maintainable. Even if something would be changed within one of the sub-views, it would not affect the aggregated view or other sub-views.To materialize or not
\nThe conclusion
\nHow We Implement Feature Requests at AUTO1
\n\n
\nMethod 1 - The Aggregator Pattern
\nMethod 2 - Importing the data into the microservice
\nTech Decision 2: Event Sourcing the State Changes
\nRefurbishment.State then the topic is named Refurbishment_State. Messages are then routed to the subscribed SQS queues using topics that have been specified. These messages can be consumed by multiple consumers.Tech Decision 3: Implementing Feature Flags
\n4. Quality Assurance (QA)
\n\n
\n
\nSoftware quality issues have an impact on functionality and as a result the user experience. When it comes to developing good software, the user always comes first. Only this ensures that the product is of high quality.
\nThose who recognize quality assurance as an essential component of software development from the start save time and money. The later errors are discovered, the longer the development time and the higher the costs.
\nContinuous quality assurance throughout the development process ensures that security vulnerabilities are detected and addressed early on.5. Monitoring
\n\n
\n\n\n
","fields":{"slug":"/how-we-manage-network-devices-at-auto1/","tags":["auto1","IT","Network","Automation "]}}},{"node":{"id":"cfee7911-e765-5a6e-ba90-1eb2ce96d837","frontmatter":{"category":"Architecture","title":"Application Cockpit Model Outline","date":"2022-10-28","summary":"A short trip around the AUTO1 Application Cockpit data model","thumbnail":null,"authorName":"Przemysław Walat","authorDescription":"Przemysław is a Expert Software Engineer based in our Szczecin office","authorAvatar":{"relativePath":"pages/application-cockpit-model-outline/avatar_Walat.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAQDAQX/xAAWAQEBAQAAAAAAAAAAAAAAAAABAAP/2gAMAwEAAhADEAAAAYNcbctZHqE5AEDf/8QAGRAAAwEBAQAAAAAAAAAAAAAAAQIDEhEA/9oACAEBAAEFAnyRJdo0aBmiW9jE1khCKBOxPVY8/8QAGBEAAgMAAAAAAAAAAAAAAAAAABEBECH/2gAIAQMBAT8Beji//8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAgEBPwEf/8QAHRAAAgIBBQAAAAAAAAAAAAAAAAERQQIgISIxgf/aAAgBAQAGPwJvpqi/CMJg4KTFN7l6P//EABsQAAMBAAMBAAAAAAAAAAAAAAABESExQWFR/9oACAEBAAE/IYU8XDGNg3v6Gar2x1WU9pKgnlpCG63tFjubo1NwXmn/2gAMAwEAAgADAAAAENfwvP/EABgRAAMBAQAAAAAAAAAAAAAAAAABEUFx/9oACAEDAQE/EHENHQ0h/8QAGREBAAIDAAAAAAAAAAAAAAAAAQAhEBFh/9oACAECAQE/EANUy+Z//8QAHRABAAIDAAMBAAAAAAAAAAAAAQARMUFRIWFxgf/aAAgBAQABPxB5rcKC843EOZKEWYB/dSvRVSN3XnMCCmUgo98hrV7b0d+xivco2sPKSmmz7hteDByVo6Gf/9k=","width":50,"height":50,"src":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg","srcSet":"/static/e4e77cd92050dbb448a088991f2d0c50/d2d31/avatar_Walat.jpeg 1x,\n/static/e4e77cd92050dbb448a088991f2d0c50/0b804/avatar_Walat.jpeg 1.5x,\n/static/e4e77cd92050dbb448a088991f2d0c50/753c3/avatar_Walat.jpeg 2x,\n/static/e4e77cd92050dbb448a088991f2d0c50/31ca8/avatar_Walat.jpeg 3x"}}},"headerImage":null},"html":"\n \n\n\n1. On the Unimus platform we add each device with the devices IP and Credentials. \n\n \n \n \n \n2. From “Schedules” we configure the time and date for the desired upgrade. \n\n \n \n \n \n3. After adding all devices of the same model to Unimus, we go to the “Mass Configuration Push” menu. Templates and configurations for the upgrade need to be prepared. \n
4. From here we also add a Mikrotik Script to download the latest update, add the schedule for this push and select the target devices.\n \n \n \n \n\n5. Lastly we add a second command to issue the upgrade and reboot the device. \n
6. All that’s left to be done is to add a corresponding downtime in our monitoring tool (CheckMK) and wait for the magic to happen.\n \n \n I’m a model, now you know what I mean
\nWhere we're at
\nResources
\n\n
\n\n
\n\n
\n\n
\n
\nReadings
\n\n
\n\n
\n
\nThe tradeoffs
\n
\nThe Resource id conundrum
\n\n
\n
\nCan I haz a Resource ?
\n
\n
\nand the (mostly) harmless evolution
\n
\nTo replace or... not to replace
\n\n
\nA little housekeeping never killed nobody... I hope
\n
\nThat's all, folks
\nMachine Learning pipeline
\nIntroduction
\n\n
\n Use Case
\nOverview
\n
\n
\n\ndef hasChanges = sh(returnStatus: true, script: \"git diff --exit-code\")\nbuild(job: 'java/core-svc/core_test_end_to_end_test', parameters: [booleanParam(name: 'SEND_SLACK_NOTIFICATION', value: \"true\"), string(name: 'RELEASE_TRAIN_VERSION', value: service_commons_dependencies_version)])\nString latestTagDateString = sh(returnStdout: true, script: \"git rev-list -1 \\$(git describe --abbrev=0 --tags) --pretty=oneline --format=%ci | awk 'NR%2==0'\")\n\ndef proceedWithE2ETesting = wasTagReleasedRecently(latestTagDateString)\n\n@NonCPS\n\n def wasTagReleasedRecently(String lastTagDateString) {\n\n def formatter = DateTimeFormatter.ofPattern(\"yyyy-MM-dd HH:mm:ss Z\")\n\n def lastTagDateTime = OffsetDateTime.parse(lastTagDateString.trim(), formatter)\n\n def currentTimeMinus2hours = OffsetDateTime.now().minus(2, ChronoUnit.HOURS)\n\n return lastTagDateTime.isAfter(currentTimeMinus2hours)\n\n}\ndefinition {\n cpsScm {\n scm {\n git {\n remote {\n github(\"core-e2e-tests\", 'ssh')\n credentials(\"token\")\n branch(\"master\")\n }\n extensions {\n wipeOutWorkspace()\n }\n }\n }\n\n scriptPath(\"jenkins/core_e2e_tests.gvy\")\n }\n}\nScenario: Should have a working API endpoint for creating a message\n\nGiven an object with generated id and message: This is a test\nWhen making a POST call on messaging endpoint with the body containing the given object in JSON representation\nThen should result in a 201 response status code\nAnd should receive an empty body\nSanity checks for new release train X.Y.Z on env:\n\n * REST API simple entity integration ✅\n * REST API dated entity integration ✅\n * REST API auditable entity integration\n * Messaging integration ✅\n * Messaging FIFO integration ✅\n * Storage integration ✅\n * Health integration ✅\n * Metrics integration ✅\n * Logging integration ✅Rate Limiters
\n\nfinal RateLimiter rateLimiter = RateLimiter.create(2.0); // rate is \"2 permits per second\"\n void submitTasks(List<Runnable> tasks, Executor executor) {\n for (Runnable task : tasks) {\n rateLimiter.acquire(); // may wait\n executor.execute(task);\n }\n}\n // Create a custom RateLimiter configuration\nRateLimiterConfig config = RateLimiterConfig.custom()\n .timeoutDuration(Duration.ofMillis(100))\n .limitRefreshPeriod(Duration.ofSeconds(1))\n .limitForPeriod(2)\n .build();\n// Create a RateLimiter\nRateLimiter rateLimiter = RateLimiter.of(**\"backendName\"**, config);\n\n// Decorate your call to BackendService.doSomething()\nSupplier<String> restrictedSupplier = RateLimiter\n .decorateSupplier(rateLimiter, backendService::doSomething);\n\n// First call is successful\nTry<String> firstTry = Try.ofSupplier(restrictedSupplier);\nassertThat(firstTry.isSuccess()).isTrue();\nRefill refill = Refill.intervally(2, Duration.ofSeconds(1));\nBandwidth limit = Bandwidth.classic(2, refill);\nBucket bucket = Bucket4j.builder()\n .addLimit(limit)\n .build();\n\nif (bucket.tryConsume(1)) {\n executor.execute(task);\n}Thread Blocking
\n\nprivate void guavaRateLimiter() {\nRateLimiter rateLimiter = RateLimiter.create(2);\nlogWithThreadTimeStamped(\"Starting\");\n\nIntStream.rangeClosed(1, 4).forEach(i -> {\n rateLimiter.acquire();\n logWithThreadTimeStamped(\"Task n. \" + i);\n});\n\nlogWithThreadTimeStamped(\"I'm blocked =(\");\nlogWithThreadTimeStamped(\"Finishing\");\n\n}\n[12:01:31.972073] Starting Thread: main\n[12:01:31.978520] Task n. 1 Thread: main\n[12:01:32.471752] Task n. 2 Thread: main\n[12:01:32.967231] Task n. 3 Thread: main\n[12:01:33.470331] Task n. 4 Thread: main\n[12:01:33.470662] I'm blocked =( Thread: main\n[12:01:33.470845] Finishing Thread: mainScheduling
\n\nprivate void taskScheduling() {\nvar scheduleExecutor = Executors.newSingleThreadScheduledExecutor();\nvar intervalDiary = new AtomicInteger();\nvar expectedInterval = 500;\nlogWithThread(\"Starting\");\n\nIntStream.rangeClosed(1, 4).forEach(i -> {\n scheduleExecutor.schedule(\n () -> logWithThread(\"Task n. \" + i),\n intervalDiary.getAndAdd(expectedInterval), TimeUnit.MILLISECONDS\n );\n});\n\nlogWithThread(\"I'm NOT blocked =D\");\nlogWithThread(\"Finishing\");\n}\n\nprivate void logWithThread(String message) {\n System.out.println(message + \" Thread: \" + Thread.currentThread().getName());\n}\nStarting - Thread: main\nI'm NOT blocked =D Thread: main\nFinishing - Thread: main\nTask n. 1 - Thread: pool-1-thread-1\nTask n. 2 - Thread: pool-1-thread-1\nTask n. 3 - Thread: pool-1-thread-1\nTask n. 4 - Thread: pool-1-thread-1\nprivate void taskSchedulingV2() {\nvar scheduleExecutor = Executors.newSingleThreadScheduledExecutor();\nvar intervalDiary = new AtomicInteger();\nvar expectedInterval = 500;\nscheduleExecutor.execute(() -> logWithThread(\"Starting\"));\n\nIntStream.rangeClosed(1, 4).forEach(i -> {\n scheduleExecutor.schedule(\n () -> logWithThread(\"Task n. \" + i),\n intervalDiary.getAndAdd(expectedInterval), TimeUnit.MILLISECONDS\n );\n});\n\nscheduleExecutor.execute(() -> logWithThread(\"I'm NOT blocked =D\"));\nscheduleExecutor.execute(() -> logWithThread(\"Finishing\"));\n}\nStarting Thread: pool-1-thread-1\nTask n. 1 Thread: pool-1-thread-1\nI'm NOT blocked =D Thread: pool-1-thread-1\nFinishing Thread: pool-1-thread-1\nTask n. 2 Thread: pool-1-thread-1\nTask n. 3 Thread: pool-1-thread-1\nTask n. 4 Thread: pool-1-thread-1\nprivate void taskSchedulingV3() {\nvar scheduleExecutor = Executors.newSingleThreadScheduledExecutor();\nvar myTasks = new ConcurrentLinkedQueue<Runnable>();\nvar EXPECTED_INTERVAL = 500;\nscheduleExecutor.execute(() -> logWithThread(\"Starting\"));\n\nscheduleExecutor.scheduleAtFixedRate(() ->\n ofNullable(myTasks.poll()).ifPresent(Runnable::run),\n 0, EXPECTED_INTERVAL, TimeUnit.MILLISECONDS\n);\n\nIntStream.rangeClosed(1, 4).forEach(i ->\n myTasks.add(() -> logWithThread(\"Task n.\" + i))\n);\n\nscheduleExecutor.execute(() -> logWithThread(\"I'm NOT blocked =D\"));\nscheduleExecutor.execute(() -> logWithThread(\"Finishing\"));\n}\nStarting Thread: pool-1-thread-1\nTask n.1 Thread: pool-1-thread-1\nI'm NOT blocked =D Thread: pool-1-thread-1\nFinishing Thread: pool-1-thread-1\nTask n.2 Thread: pool-1-thread-1\nTask n.3 Thread: pool-1-thread-1\nTask n.4 Thread: pool-1-thread-1\nprivate void taskSchedulingOverload() {\nvar scheduleExecutor = Executors.newSingleThreadScheduledExecutor();\nvar myTasks = new ConcurrentLinkedQueue<Runnable>();\nvar EXPECTED_INTERVAL = 500;\nvar SMALLER_INTERVAL = 250;\nscheduleExecutor.execute(() -> logWithThread(\"Starting\"));\n\nscheduleExecutor.scheduleAtFixedRate(() ->\n ofNullable(myTasks.poll()).ifPresent(Runnable::run),\n 0, EXPECTED_INTERVAL, TimeUnit.MILLISECONDS\n);\n\n\nscheduleExecutor.scheduleAtFixedRate(() ->\n myTasks.add(() -> logWithThread(\"Fast Task! Task lasting: \" + myTasks.size())),\n 0, SMALLER_INTERVAL, TimeUnit.MILLISECONDS\n);\n\n\nscheduleExecutor.execute(() -> logWithThread(\"I'm NOT blocked =D\"));\nscheduleExecutor.execute(() -> logWithThread(\"Finishing\"));\n}\nStarting Thread: pool-1-thread-1\nI'm NOT blocked =D Thread: pool-1-thread-1\nFinishing Thread: pool-1-thread-1\nFast Task! Task lasting: 1 Thread: pool-1-thread-1\nFast Task! Task lasting: 2 Thread: pool-1-thread-1\nFast Task! Task lasting: 3 Thread: pool-1-thread-1\nFast Task! Task lasting: 4 Thread: pool-1-thread-1\nFast Task! Task lasting: 5 Thread: pool-1-thread-1\nFast Task! Task lasting: 6 Thread: pool-1-thread-1\nFast Task! Task lasting: 7 Thread: pool-1-thread-1\nFast Task! Task lasting: 8 Thread: pool-1-thread-1\nFast Task! Task lasting: 9 Thread: pool-1-thread-1\n\n(...)\nprivate void taskSchedulingLimitingByQueueSize() {\nvar scheduleExecutor = Executors.newSingleThreadScheduledExecutor();\nvar myTasks = new ConcurrentLinkedQueue<Runnable>();\nvar EXPECTED_INTERVAL = 500;\nvar SMALLER_INTERVAL = 250;\nvar QUEUE_LIMIT = 4;\nscheduleExecutor.execute(() -> logWithThread(\"Starting\"));\n\nscheduleExecutor.scheduleAtFixedRate(() ->\n ofNullable(myTasks.poll()).ifPresent(Runnable::run),\n 0, EXPECTED_INTERVAL, TimeUnit.MILLISECONDS\n);\n\n\nscheduleExecutor.scheduleAtFixedRate(() -> {\n if (myTasks.size() < QUEUE_LIMIT)\n myTasks.add(() -> logWithThread(\"Fast Task! Task lasting: \" + myTasks.size()));\n else\n logWithThread(\"I was rejected =(\");\n },\n 0, SMALLER_INTERVAL, TimeUnit.MILLISECONDS\n);\n\nscheduleExecutor.execute(() -> logWithThread(\"I'm NOT blocked =D\"));\nscheduleExecutor.execute(() -> logWithThread(\"Finishing\"));\n}\nStarting Thread: pool-1-thread-1\nI'm NOT blocked =D Thread: pool-1-thread-1\nFinishing Thread: pool-1-thread-1\nFast Task! Task lasting: 1 Thread: pool-1-thread-1\nFast Task! Task lasting: 2 Thread: pool-1-thread-1\nFast Task! Task lasting: 3 Thread: pool-1-thread-1\nI was rejected =( Thread: pool-1-thread-1\nFast Task! Task lasting: 3 Thread: pool-1-thread-1\nI was rejected =( Thread: pool-1-thread-1\nFast Task! Task lasting: 3 Thread: pool-1-thread-1\nI was rejected =( Thread: pool-1-thread-1\nFast Task! Task lasting: 3 Thread: pool-1-thread-1Conclusion
\nEfficient Testing - Automate Early
\nPlanning and reconnaissance
\nTest data preparation
\nPage Object skeletons
\n![](\"https://i.postimg.cc/tXm79NqD/image.png\")
Doubts?
\nIn real life
\nConclusion
\nLogging with MDC in functional style
\nIntroduction
\nUse case
\nMDC code drawbacks
\n\ntry {\n MDC.put(\"key1\", \"value1\");\n MDC.put(\"key1\", \"value2\");\n //some process\n log.info(\"very important log\");\n}finally {\n MDC.remove(\"key1\");\n MDC.remove(\"key2\");\n}method putCloseable(key, value) which returns AutoClosable object but it allows only one key-value pair and has to be assigned to a variable, which may look better for a single key but it is not perfect. IDE will complain about never used variable\ntry (MDC.MDCCloseable closeable = MDC.putCloseable(\"key\", \"value\")) {\n //some process\n log.info(\"very important log\");\n}Solution
\n\nLogContext.withLogKey(\"key1\", \"value1\")\n .andKey(\"key2\", \"value2\")\n .andKey(\"key3\", \"value3\")\n .execute(() -> {\n // some process\n log.info(\"very important log\");\n });LogContextSteps
\n\n@NoArgsConstructor(access = AccessLevel.PRIVATE)\npublic final class LogContextSteps {\n\n public interface ContextStep {\n ExecuteStep andKey(final String key, final Object value);\n }\n\n public interface ExecuteStep extends ContextStep {\n\n <T> T execute(final Supplier<T> action);\n\n default void execute(final Runnable action) {\n if (action == null) {\n throw new IllegalArgumentException(\"Action for execution cannot be null.\");\n }\n\n execute(() -> {\n action.run();\n return null;\n });\n }\n }\n}\n
\nLogContext
\n\npublic class LogContext implements ContextStep, ExecuteStep {\n private static final MDCLogContextSetter MDC_LOG_CONTEXT_SETTER = new MDCLogContextSetter();\n\n private final LogContextSetter contextSetter;\n private final Set<String> contextKeys = new HashSet<>();\n\n private LogContext(final LogContextSetter contextSetter) {\n this.contextSetter = contextSetter;\n }\n\n public static ExecuteStep withLogKey(String key, Object value) {\n return withContextSetter(MDC_LOG_CONTEXT_SETTER).andKey(key,value);\n }\n\n static ExecuteStep withContextSetter(LogContextSetter contextSetter){\n return new LogContext(contextSetter);\n }\n\n @Override\n public ExecuteStep andKey(String key, Object value) {\n contextSetter.put(key, value);\n contextKeys.add(key);\n return this;\n }\n\n @Override\n public <T> T execute(final Supplier<T> action) {\n if (action == null) {\n throw new IllegalArgumentException(\"Action for execution cannot be null.\");\n }\n\n try {\n return action.get();\n } finally {\n contextKeys.forEach(contextSetter::remove);\n }\n }\n\n interface LogContextSetter {\n\n void remove(final String key);\n\n void put(final String key, final Object value);\n }\n\n private static class MDCLogContextSetter implements LogContextSetter {\n\n @Override\n public void remove(String key) {\n MDC.remove(key);\n }\n\n @Override\n public void put(String key, Object value) {\n MDC.put(key, Objects.toString(value));\n }\n }\n}LogContext is our main class here. It:\n
\nLogContextStepsSetLogContextSetter interface and its implementation MDCLogContextSetter, as it would be hard to test static methods of MDC class.LogContext.withLogKey(\"key1\", \"value1\") which sets up our MDCLogContextSetter and invokes the andKey(\"key2\", \"value2\") method for the first time. Then we can chain next invocations of andKey where every invocation causes two things:\n
\nSet<String> contextKeys as only keys are needed to clear contextLogContextSetter::put method is invoked where implementation is: MDC.put(key, Objects.toString(value)) and at that moment key-value is stored in MDCexecute(() -> action()) which is classic try-finally block where try invokes our action and finally clears MDC iterating every through contextKeys and invoking LogContextSetter::remove where implementation id MDC.remove(key)Practical usage (filtering in Kibana)
\nMacBook with ARM chip and embedded-libraries problems
\nEmbedded-postgres problems
\nProblem
\n\n Factory method 'postgres' threw exception; nested exception is\n java.lang.IllegalStateException: Process\n [/var/folders/61/ls8bv20x4w3fsn3wpm8dlpfm0000gp/T/embedded-pg/\n PG-0c0585587b40695247cf72650cd42f92/bin/initdbSolution
\n\nsudo sysctl kern.sysv.shmmax=104857600\nsudo sysctl kern.sysv.shmall=25600\nsudo sysctl kern.sysv.shmmin=1\nsudo sysctl kern.sysv.shmmni=10240\nsudo sysctl kern.sysv.shmseg=4096\n\nexport LC_NUMERIC=en_US.UTF-8\nexport LC_MONETARY=en_US.UTF-8\nexport LC_TIME=en_US.UTF-8\nexport LC_ALL=en_US.UTF-8\nexport LC_COLLATE=en_US.UTF-8\nexport LC_MESSAGES=en_US.UTF-8\nexport LC_CTYPE=en_US.UTF-8\nexport LANG=en_US.UTF-8Embedded-redis problems
\nProblem
\n\njava.lang.RuntimeException: Can't start redis server. Check\nlogs for details.\n\nat\nredis.embedded.AbstractRedisInstance.awaitRedisServerReady\n(AbstractRedisInstance.java:61)\nat\nredis.embedded.AbstractRedisInstance.start\n(AbstractRedisInstance.java:39)\nat redis.embedded.RedisServer.start(RedisServer.java:9)\nat\nwkda.common.distributed.cache.redis.EmbeddedSingleRedisServer\n.before(EmbeddedSingleRedisServer.java:48)\nat org.springframework.test.context.junit4.SpringJUnit4ClassRunner\n.run(SpringJUnit4ClassRunner.java:190)Deep diving into problem
\n\n=== REDIS BUG REPORT START: Cut & paste starting from here ===\n30203:M 15 Feb 14:01:14.039 # Redis 3.2.4 crashed by signal: 10\n30203:M 15 Feb 14:01:14.040 # Crashed running the instuction\nat: 0x7fff20406440\n30203:M 15 Feb 14:01:14.040 # Accessing address: 0x305685000\n30203:M 15 Feb 14:01:14.040 # Failed assertion: <no\nassertion failed> (<no file>:0)\n\n------ STACK TRACE ------\nEIP:\n0 libsystem_platform.dylib 0x00007fff20406440\n_platform_memset$VARIANT$Rosetta + 108\n\nBacktrace:\n0 redis-server.app 0x0000000100a70a7e logStackTrace + 110\n1 redis-server.app 0x0000000100a70e4d sigsegvHandler + 253\n2 libsystem_platform.dylib 0x00007fff20403d7d _sigtramp + 29\n3 ??? 0x00000001091c1200 0x0 + 4447801856\n4 ??? 0x31207070612e7265 0x0 + 3539952935082291813Solution
\n\n
\n\nsysctl -a | grep hw.optional\n\n------- Result -------\n\nhw.optional.amx_version: 2\nhw.optional.arm64: 1\nhw.optional.armv8_1_atomics: 1\nhw.optional.armv8_2_fhm: 1\nhw.optional.armv8_2_sha3: 1\nhw.optional.armv8_2_sha512: 1\nhw.optional.armv8_crc32: 1\nhw.optional.breakpoint: 6\nhw.optional.floatingpoint: 1\nhw.optional.neon: 1\nhw.optional.neon_fp16: 1\nhw.optional.neon_hpfp: 1\nhw.optional.ucnormal_mem: 1\nhw.optional.watchpoint: 4\nsysctl -a | grep hw.optional\nhw.optional.floatingpoint: 1\nhw.optional.mmx: 1\nhw.optional.sse: 1\nhw.optional.sse2: 1\nhw.optional.sse3: 1\nhw.optional.supplementalsse3: 1\nhw.optional.sse4_1: 1\nhw.optional.sse4_2: 1\nhw.optional.x86_64: 1\nhw.optional.aes: 1\nhw.optional.avx1_0: 1\nhw.optional.rdrand: 1\nhw.optional.f16c: 1\nhw.optional.enfstrg: 1\nhw.optional.fma: 1\nhw.optional.avx2_0: 1\nhw.optional.bmi1: 1\nhw.optional.bmi2: 1\nhw.optional.rtm: 0\nhw.optional.hle: 0\nhw.optional.adx: 1\nhw.optional.mpx: 0\nhw.optional.sgx: 0\nhw.optional.avx512f: 0\nhw.optional.avx512cd: 0\nhw.optional.avx512dq: 0\nhw.optional.avx512bw: 0\nhw.optional.avx512vl: 0\nhw.optional.avx512ifma: 0\nhw.optional.avx512vbmi: 0\nprotected void before() {\nRedisExecProvider provider = RedisExecProvider.defaultProvider()\n.override(OS.UNIX, \"redis-server/redis-server\")\n.override(OS.WINDOWS, Architecture.x86,\n\"redis-server/redis-server.exe\")\n.override(OS.WINDOWS, Architecture.x86_64,\n\"redis-server/redis-server.exe\")\n.override(OS.MAC_OS_X, Architecture.x86,\n\"redis-server/redis-server.app\")\n.override(OS.MAC_OS_X, Architecture.x86_64,\ngetRedisServerForAppleArchitecture());\n\n\nthis.redisServer = RedisServer.builder()\n.redisExecProvider(provider)\n.port(redisPort)\n.setting(\"bind \" + this.redisHost)\n.build();\n\nthis.redisServer.start();\nlog.info(\"Started Redis server at [{}:{}]\",\nthis.redisHost, this.redisPort);\n}\n\n\nprivate String getRedisServerForAppleArchitecture()\nthrows Exception {\nProcess process = Runtime.getRuntime().exec(\"sysctl hw\");\ntry (BufferedReader input = new BufferedReader(new\nInputStreamReader(process.getInputStream()))) {\nString line;\nwhile((line = input.readLine()) != null) {\nif (line.contains(\"hw.optional.arm64\") &&\nline.trim().endsWith(\"1\")) {\nreturn \"redis-server/amd64/redis-server.app\";\n}\n}\n} catch (IOException e) {\nlog.error(\"Error during parsing system process\");\nthrow new RuntimeException(e);\n}\nreturn \"redis-server/redis-server.app\";\n}\n
\n\n
\n\n
\n\n
\n\n
\n\n
\nThe OpenAPI journey
\nThe situation
\nThe light at the end of the tunnel
\nDesign first, code later
\nThe ecosystem
\nRetrofitting OpenAPI
\nChallenges ahead
\nThe verdict
\nWhat is a Test Strategy?
\n\n
\nWhy do we create a Test Strategy?
\nHow do we create a Test Strategy?
\nScope of the testing
\n\n
\nRisks
\n\n
\nAutomation strategy
\n\n
\nRequired test data
\nSummary
\n\n
\n\nmy_gradle_plugin\n\n├── build.gradle\n\n├── gradle\n\n│ └── wrapper\n\n│ ├── gradle-wrapper.jar\n\n│ └── gradle-wrapper.properties\n\n├── gradlew\n\n├── gradlew.bat\n\n├── settings.gradle\n\n└── src\n\n├── main\n\n│ ├── java\n\n│ └── resources\n\n│ └── META-INF\n\n│ └── plugin.xml\n\n└── test\n\n├── java\n\n└── resources\n\n
\nGOAL 2 - As a user I want to see the result notification.\n\n<actions>\n\n<action id=\"GeneratePOJOAction\" class=\"GeneratePOJOAction\"\n\nicon=\"/icons/pojo.png\"\n\ntext=\"Generate POJO from JSON\"\n\ndescription=\"GeneratePOJOAction\">\n\n<add-to-group group-id=\"NewGroup\" anchor=\"last\"/>\n\n</action>\n\n</actions>\nimport com.intellij.openapi.actionSystem.AnAction\n\nimport com.intellij.openapi.actionSystem.AnActionEvent\n\nclass GeneratePOJOAction : AnAction() {\n\noverride fun actionPerformed(e: AnActionEvent) { \n\n//TODO: Add your UI implementation here \n\n} \n}\ncompanion object {\n\nprivate const val BUTTON_TITLE = \"OK\"\n\nprivate const val DEFAULT_OPTION_INDEX = -1\n\nprivate val DEFAULT_ICON = null\n\n}\n\nfun onPluginExceptionHandled(exception: RoboPluginException) {\n\nMessages.showDialog(\n\nexception.message, exception.title,\n\narrayOf(BUTTON_TITLE),\n\nDEFAULT_OPTION_INDEX,\n\nDEFAULT_ICON\n\n)\n\n}\n12:00 Response.java was updated.\n \n12:00 POJO generation: Success\ncompanion object{\n\nprivate const val MESSAGE_SUCCESS = \"POJO generation: Success\"\n\nprivate const val GROUP_ID = \"RoboPOJO Generator\"\n\n}\n\nfun showSuccessMessage() {\n\nshowMessage(\n\nproject = ProjectManager.getInstance().openProjects.first(),\n\nmessage = MESSAGE_SUCCESS,\n\ntype = NotificationType.INFORMATION,\n\n)\n\n}\n\nprivate fun showMessage(\n\nmessage: String, type: NotificationType, project: Project\n\n) = NotificationGroupManager.getInstance()\n\n.getNotificationGroup(GROUP_ID)\n\n.createNotification(message, type)\n\n.notify(project)\n\n \n<extensions defaultExtensionNs=\"com.intellij\">\n\n<notificationGroup id=\"RoboPOJO Generator\"\n\ndisplayType=\"BALLOON\"\n\nkey=\"com.robohorse.robopojogenerator\"/>\n\n</extensions>The problems
\nHeadcount allocation
\nGoogle spreadsheet UI
\n\n
\nWorkflow process
\n\n
\nLack of automation
\nThe answer
\n\n
\n\n
\nHow it works
\nSo, how does it help us?
\nTech allocation
\n![](\"https://lh4.googleusercontent.com/EZIHgD1u6-uVAG_c0AW2Y4QEf6Hia0Wt1Y4QMRaOc5y31Qb1Qg2M6JuWvnwZwTBh1OoIzJ-7BhLJMF0hjRM767YP405F25mx0KF3k6OH2EOJzeKl_0TeKQIlOQDVo61qUjL9M7qD\")
Days invested
\nA few more benefits
\n\n
\n\n
\n\n
","fields":{"slug":"/TechOS/","tags":["auto1","engineering","projectmanagement"]}}},{"node":{"id":"a5e0a01f-3e0b-578a-9e70-98d511e929fb","frontmatter":{"category":"TechRadar","title":"AUTO1 Tech Radar October 2020","date":"2020-10-09","summary":"AUTO1 Tech Radar October 2020","thumbnail":{"relativePath":"pages/tech-radar-october-2020/radar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAECBf/EABcBAAMBAAAAAAAAAAAAAAAAAAECAwD/2gAMAwEAAhADEAAAAePlJzgZgw0FH//EABsQAAIBBQAAAAAAAAAAAAAAAAABAhAREjFB/9oACAEBAAEFAoxyozg9lxn/xAAWEQADAAAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BqP/EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EABUQAQEAAAAAAAAAAAAAAAAAACEw/9oACAEBAAY/AmP/xAAZEAADAQEBAAAAAAAAAAAAAAAAAREhMRD/2gAIAQEAAT8hsuCK6ht0TxDvlZwsRXh//9oADAMBAAIAAwAAABCYMEP/xAAYEQADAQEAAAAAAAAAAAAAAAAAAREhMf/aAAgBAwEBPxDaLhBnD//EABgRAAMBAQAAAAAAAAAAAAAAAAABESEx/9oACAECAQE/ENouEE8P/8QAGxABAAMBAQEBAAAAAAAAAAAAAQARIVFBMYH/2gAIAQEAAT8QG6DVB5EKCJzyIMp+AR7l8xiq6sRgprIurxFaML5CYBP/2Q==","width":326,"height":325,"src":"/static/38ec33b907e70d89d23101b34889b707/a2998/radar.jpg","srcSet":"/static/38ec33b907e70d89d23101b34889b707/a2998/radar.jpg 1x,\n/static/38ec33b907e70d89d23101b34889b707/99cce/radar.jpg 1.5x,\n/static/38ec33b907e70d89d23101b34889b707/6e995/radar.jpg 2x,\n/static/38ec33b907e70d89d23101b34889b707/968b1/radar.jpg 3x"}}},"authorName":"AUTO1 Group","authorDescription":"AUTO1 Group","authorAvatar":{"relativePath":"pages/tech-radar-october-2020/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/tech-radar-october-2020/radar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAECBf/EABcBAAMBAAAAAAAAAAAAAAAAAAECAwD/2gAMAwEAAhADEAAAAePlJzgZgw0FH//EABsQAAIBBQAAAAAAAAAAAAAAAAABAhAREjFB/9oACAEBAAEFAoxyozg9lxn/xAAWEQADAAAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BqP/EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EABUQAQEAAAAAAAAAAAAAAAAAACEw/9oACAEBAAY/AmP/xAAZEAADAQEBAAAAAAAAAAAAAAAAAREhMRD/2gAIAQEAAT8hsuCK6ht0TxDvlZwsRXh//9oADAMBAAIAAwAAABCYMEP/xAAYEQADAQEAAAAAAAAAAAAAAAAAAREhMf/aAAgBAwEBPxDaLhBnD//EABgRAAMBAQAAAAAAAAAAAAAAAAABESEx/9oACAECAQE/ENouEE8P/8QAGxABAAMBAQEBAAAAAAAAAAAAAQARIVFBMYH/2gAIAQEAAT8QG6DVB5EKCJzyIMp+AR7l8xiq6sRgprIurxFaML5CYBP/2Q==","width":1280,"height":1278,"src":"/static/38ec33b907e70d89d23101b34889b707/966a5/radar.jpg","srcSet":"/static/38ec33b907e70d89d23101b34889b707/966a5/radar.jpg 1x,\n/static/38ec33b907e70d89d23101b34889b707/aa36d/radar.jpg 1.5x,\n/static/38ec33b907e70d89d23101b34889b707/26d0b/radar.jpg 2x,\n/static/38ec33b907e70d89d23101b34889b707/39d45/radar.jpg 3x"}}}},"html":"","fields":{"slug":"/tech-radar-october-2020/","tags":["TechRadar"]}}},{"node":{"id":"dc3aa441-f1ec-5a7d-b881-e2c93b86ddfd","frontmatter":{"category":"DevOps","title":"Improving our CI/CD system performance with storage migration","date":"2020-09-21","summary":"Leveraging AWS Elastic Block Store for improved performance and faster build times.","thumbnail":null,"authorName":"Rafael Silva","authorDescription":"Rafael is a DevOps Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/storage-system-improvements/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAMEBQb/xAAVAQEBAAAAAAAAAAAAAAAAAAAAAf/aAAwDAQACEAMQAAAB0IK9eXdVyc/CGqD/xAAcEAACAwADAQAAAAAAAAAAAAABAgADEgQREyL/2gAIAQEAAQUCY9LSxJl92J7ZCPtLT9zjk4//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/AR//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAcEAEAAQQDAAAAAAAAAAAAAAABAAIQESExQYH/2gAIAQEABj8CcczatsU9waXcGNvZ/8QAHBABAAMAAgMAAAAAAAAAAAAAAQARMRAhQVFh/9oACAEBAAE/IWXgdTa57OHBUur8SwRbZ9x2IyIB0x//2gAMAwEAAgADAAAAECAnvv/EABYRAQEBAAAAAAAAAAAAAAAAAAERIP/aAAgBAwEBPxAkcf/EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8QH//EAB0QAQACAQUBAAAAAAAAAAAAAAEAESExQVFxgaH/2gAIAQEAAT8Qrrsti8xYDhdIOK+y5TBSrKnUFnZQl1zfsIcLGBs7kuV0QImCShbHhP/Z","width":50,"height":50,"src":"/static/2eae20fca46dd64af3b25077e9ded172/45876/avatar.png","srcSet":"/static/2eae20fca46dd64af3b25077e9ded172/45876/avatar.png 1x,\n/static/2eae20fca46dd64af3b25077e9ded172/eb85b/avatar.png 1.5x,\n/static/2eae20fca46dd64af3b25077e9ded172/4f71c/avatar.png 2x,\n/static/2eae20fca46dd64af3b25077e9ded172/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"Improving our CI/CD system performance with storage migration
\nPerformance improvements with EBS
\nReferences
\n\n<dependency> \n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-batch</artifactId>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-data-jpa</artifactId>\n</dependency> \n\n<dependency>\n <groupId>org.projectlombok</groupId>\n <artifactId>lombok</artifactId>\n <version>1.18.10</version>\n <scope>provided</scope>\n</dependency>\n\n<dependency>\n <groupId>com.h2database</groupId>\n <artifactId>h2</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.batch</groupId>\n <artifactId>spring-batch-test</artifactId>\n <scope>test</scope>\n</dependency>\n\n<dependency>\n <groupId>org.hamcrest</groupId>\n <artifactId>hamcrest-all</artifactId>\n <version>1.3</version>\n <scope>test</scope>\n</dependency>\n@Data\n\npublic class Book {\nprivate String title;\nprivate String description;\nprivate String author;\n}\n@Configuration\n@EnableBatchProcessing\n@RequiredArgsConstructor\npublic class SpringBatchConfiguration\nprivate final JobBuilderFactory jobBuilderFactory;\nprivate final StepBuilderFactory stepBuilderFactory;\n@Bean\n@StepScope\npublic FlatFileItemReader<Book> bookReader(@Value(\"#{jobParameters['filePath']}\") final String filePath\n) {\nreturn new FlatFileItemReaderBuilder<Book>()\n .name(\"personItemReader\")\n .resource(new ClassPathResource(filePath))\n .delimited()\n .names(new String[]{\"title\", \"description\", \"author\"})\n .fieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {{\n setTargetType(Book.class);\n }})\n .build();\n}\n@Bean\npublic JdbcBatchItemWriter<Book> writer(final DataSource dataSource) {\n return new JdbcBatchItemWriterBuilder<Book>()\n .itemPreparedStatementSetter((book, preparedStatement) -> {\n preparedStatement.setString(1, book.getTitle());\n preparedStatement.setString(2, book.getDescription());\n preparedStatement.setString(3, book.getAuthor());\n })\n .sql(\"INSERT INTO books (title, description, author) VALUES (title, description, author_surname)\")\n .dataSource(dataSource)\n .build();\n} \n@Bean\npublic Step step1(final ItemWriter<Book> writer, final ItemReader<Book> reader) {\n return stepBuilderFactory.get(\"step1\")\n .<Book, Book> chunk(100)\n .reader(reader)\n .processor((ItemProcessor<Book, Book>) book -> book)\n .writer(writer)\n .build();\n}\n@Bean\npublic Job importUserJob(final Step step1) {\n return jobBuilderFactory.get(\"bookReaderJob\")\n .incrementer(new RunIdIncrementer())\n .flow(step1)\n .end()\n .build();\n}spring.batch.job.enabled=false in your properties config file so that spring batch job won’t be executed automatically with no parameters when the application is started.\n@Component\n@RequiredArgsConstructor\npublic class SpringBatchExecutor { \n private final JobLauncher jobLauncher;\n private final Job job; \n\n @SneakyThrows\n public void execute(final String filePath) {\n JobParameters parameters = new JobParametersBuilder()\n .addString(\"filePath\", filePath)\n .toJobParameters(); \n jobLauncher.run(job, parameters);\n } \n}\n@Data\n@Entity\n@Table(name = \"books\")\npublic class BookEntity {\n @Id\n @GeneratedValue(strategy = GenerationType.IDENTITY)\n private Long id;\n private String title;\n private String description;\n private String authorFullName;\n}\n@Repository\npublic interface BookRepository extends JpaRepository<BookEntity, Long> {\n} \n@Slf4j\n\n@SpringBootTest\nclass SpringBatchSampleApplicationIntegrationTest {\n\n @Autowired\n private SpringBatchExecutor springBatchExecutor;\n \n @Autowired\n private BookRepository bookRepository;\n\n @Test\n public void testExecution() {\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"sample-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(7L));\n } \n\n @Test\n public void testLargeData() {\n long startTime = System.currentTimeMillis();\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"large-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(60000L));\n long endTime = System.currentTimeMillis();\n log.info(\"executed in miles: {}\", endTime - startTime);\n }\n}Background
\nPlugin inspections
\n\n
\n@DirtiesContext annotation which slows tests down because it forces creation of new Spring context@ActiveProfiles (slows tests down because different Spring context would have to be created for each combination)@ActiveProfiles is greater than 3@FeignClient (or DTO used by client) not documented with Swagger annotations@Transactional@DirtiesContext inspection.Create plugin
\nInspection
\norg.springframework.test.annotation.DirtiesContext and provide quick fix which deletes @DirtiesContext. If Spring is not your thing then any other annotation can be used instead.com.intellij.codeInspection.LocalInspectionTool or one of subclasses.\nGlobal inspections work only in batch mode when analysis is manually triggered via Analyze > Inspect Code and see complete graph of references between classes and can report problem for any file. Inspection class has to extend com.intellij.codeInspection.GlobalInspectionTool or one of subclasses.LocalInspectionTool and take a look what other inspections are extending. In this case AbstractBaseJavaLocalInspectionTool seems to be a good choice since most of Java inspections are based on it.DirtiesContextInspection with following content:\npublic class DirtiesContextFirstVersionInspection extends AbstractBaseJavaLocalInspectionTool {\n private static final String DIRTIES_CONTEXT = \"org.springframework.test.annotation.DirtiesContext\";\n\n private static final String DESCRIPTION_TEMPLATE = \"Usage of @DirtiesContext makes integration tests slower\";\n\n public String getDisplayName() {\n return \"Usage of @DirtiesContext is not recommended\";\n }\n\n public String getGroupDisplayName() {\n return GroupNames.PERFORMANCE_GROUP_NAME;\n }\n\n public String getShortName() {\n return \"DirtiesContext\";\n }\n\n public boolean isEnabledByDefault() {\n return true;\n }\n\n public PsiElementVisitor buildVisitor(@NotNull final ProblemsHolder holder, \n boolean isOnTheFly) {\n return new JavaElementVisitor() {\n\n public void visitAnnotation(PsiAnnotation annotation) {\n super.visitAnnotation(annotation);\n\n String qualifiedName = annotation.getQualifiedName();\n\n if (DIRTIES_CONTEXT.equals(qualifiedName)) {\n holder.registerProblem(annotation, DESCRIPTION_TEMPLATE);\n }\n }\n };\n }\n}plugin.xml which among other things describes what plugin does, what other plugins are required and in which version of IntelliJ it can be used.plugin.xml under extensions tag and configure inspection using xml but we find it easier to register inspectionToolProvider and configure using it in Java code.\npublic class CodeInspectionProvider implements InspectionToolProvider {\n public Class[] getInspectionClasses() {\n return new Class[]{\n DirtiesContextInspection.class\n };\n }\n}plugin.xml:\n <extensions defaultExtensionNs=\"com.intellij\">\n <inspectionToolProvider implementation=\"com.auto1.intellij.tutorial.CodeInspectionProvider\"/>\n </extensions>./gradlew runIde but it’s better to use Gradle view in IntelliJ since it allows to start IDE in debug mode (if needed just right click on task and select Debug, you can create run configuration to speed-up in the future). New IntelliJ instance should show up, if you already have sources of some project that uses Spring Boot then open it, otherwise you could create a new project. Annotate some class with @DirtiesContext annotation and observe inspection marker showing up. In case of problems logs can be found at build/idea-sandbox/system/log/.Adding quick fix
\nDirtiesContextInspection class and add quick fix:\n private static class DeleteQuickFix implements LocalQuickFix {\n public String getName() {\n return \"Removes usage of @DirtiesContext\";\n }\n\n public void applyFix(@NotNull Project project, @NotNull ProblemDescriptor descriptor) {\n descriptor.getPsiElement().delete();\n }\n\n public String getFamilyName() {\n return getName();\n }\n }\nholder.registerProblem(annotation, DESCRIPTION_TEMPLATE, new DeleteQuickFix());runIde Gradle task should prove that quick fix works as expected. Internationalization
\nPluginBundle class:\npublic class PluginBundle {\n private static Reference<ResourceBundle> ourBundle;\n\n private static final String BUNDLE = \"com.auto1.intellij.tutorial.PluginBundle\";\n\n private PluginBundle() { }\n\n public static String message(@NotNull @PropertyKey(resourceBundle = BUNDLE) String key, \n @NotNull Object... params) {\n return CommonBundle.message(getBundle(), key, params);\n }\n\n private static ResourceBundle getBundle() {\n ResourceBundle bundle = com.intellij.reference.SoftReference.dereference(ourBundle);\n if (bundle == null) {\n bundle = ResourceBundle.getBundle(BUNDLE);\n ourBundle = new SoftReference<>(bundle);\n }\n return bundle;\n }\n}BUNDLE constant, in my case it will be src/main/resources/com/auto1/intellij/tutorial/PluginBundle.properties with content:\ninspection.dirties.context.display.name=Usage of @DirtiesContext is not recommended\ninspection.dirties.context.problem.descriptor=Usage of @DirtiesContext makes integration tests slower\ninspection.dirties.context.use.quickfix=Removes usage of @DirtiesContextplugin.xml:\n<resource-bundle>com.auto1.intellij.tutorial.PluginBundle</resource-bundle>\n public String getDisplayName() {\n return PluginBundle.message(\"inspection.dirties.context.display.name\");\n }Testing
\nLightPlatformCodeInsightFixtureTestCase as base for our tests because it is recommended in the documentation. Unfortunately, testing appeared harder to set up properly than expected.\n protected LightProjectDescriptor getProjectDescriptor() {\n return new LightProjectDescriptor() {\n public Sdk getSdk() {\n return JavaAwareProjectJdkTableImpl.getInstanceEx().getInternalJdk();\n }\n };\n }org.springframework.test.annotation.DirtiesContext, it got DirtiesContext while it worked fine for real project in IDE. It turns out that such behaviour occurs when test project doesn’t see definition of some class. This is fixable by either hardcoding problematic class into test or by adding dependency as library to the project. Second approach avoids copying source code from other projects and seems to be more interesting so it will be presented here. In order to download dependency jar we use ShrinkWrap library:\nprivate File[] getMavenArtifacts(String... mavenArtifacts) {\n File[] files = Maven.resolver()\n .resolve(mavenArtifacts)\n .withoutTransitivity()\n .asFile();\n if (files.length == 0) {\n throw new IllegalArgumentException(\"Failed to resolve artifacts \" + Arrays.toString(mavenArtifacts));\n }\n return files;\n }\n protected void attachMavenLibrary(String mavenArtifact) {\n File[] jars = getMavenArtifacts(mavenArtifact);\n Arrays.stream(jars).forEach(jar -> {\n String name = jar.getName();\n PsiTestUtil.addLibrary(myFixture.getProjectDisposable(), myModule, name, jar.getParent(), name);\n });\n }myFixture.getProjectDisposable() instead of myFixture.getProject() otherwise there is an exception during test shutdown:\ncom.intellij.openapi.util.TraceableDisposable$DisposalException: Virtual pointer 'jar:///somePath/.m2/repository/org/springframework/spring-test/5.1.5.RELEASE/spring-test-5.1.5.RELEASE.jar!/' hasn't been disposedgetTestDataPath. Since input files most likely won’t compile because of missing imports and possible usage of special markers like src/test/java folder to store them:\n @Override\n protected String getTestDataPath() {\n return \"src/test/testData\";\n }configureByFile to load input file, doHighlighting to trigger source code analysis, launchAction to execute quick fix and finally checkResultByFile to compare results against after file.\n myFixture.configureByFile(testName + \".java\");\n myFixture.enableInspections(new DirtiesContextInspection());\n myFixture.doHighlighting();\n IntentionAction quickFixAction = myFixture.findSingleIntention(intentionHint);\n myFixture.launchAction(quickFixAction);\n myFixture.checkResultByFile(testName + \".after.java\");Next steps
\nLinks
\n\n
","fields":{"slug":"/intellij-plugin/","tags":["intellij"]}}},{"node":{"id":"e2563427-db60-5c74-aaff-0c8c2c483f69","frontmatter":{"category":"Engineering","title":"DevoxxUA was awesome!","date":"2019-12-05","summary":"A few words about one of the biggest Java conferences and AUTO1's presence there.","thumbnail":{"relativePath":"pages/auto1-devoxx/ivan_thumb.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAABQABBP/EABUBAQEAAAAAAAAAAAAAAAAAAAEA/9oADAMBAAIQAxAAAAFobuDFqyL/xAAaEAABBQEAAAAAAAAAAAAAAAACAAEDEzES/9oACAEBAAEFApMKYRldXWI3bvF//8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BJ//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8BP//EAB0QAAEEAgMAAAAAAAAAAAAAAAABERIhAxBCUYH/2gAIAQEABj8C9H496SmExxqQx//EABsQAAMAAgMAAAAAAAAAAAAAAAABETFRIUGh/9oACAEBAAE/IWihpe8JGzIwO65PagSFoof/2gAMAwEAAgADAAAAEEgv/8QAFREBAQAAAAAAAAAAAAAAAAAAEDH/2gAIAQMBAT8Qg//EABYRAQEBAAAAAAAAAAAAAAAAAAEQIf/aAAgBAgEBPxBNn//EAB4QAQACAgIDAQAAAAAAAAAAAAEAESFBMWFRcbHh/9oACAEBAAE/EPcQfYt7YYNab5CuSAbFIzJ3xnp6mFDht8n9YZ1dVqq6n//Z","width":520,"height":325,"src":"/static/150f8a5d8b98da0a021f785cc45b217a/a2998/ivan_thumb.jpg","srcSet":"/static/150f8a5d8b98da0a021f785cc45b217a/a2998/ivan_thumb.jpg 1x,\n/static/150f8a5d8b98da0a021f785cc45b217a/99cce/ivan_thumb.jpg 1.5x,\n/static/150f8a5d8b98da0a021f785cc45b217a/6e995/ivan_thumb.jpg 2x,\n/static/150f8a5d8b98da0a021f785cc45b217a/968b1/ivan_thumb.jpg 3x"}}},"authorName":"Ivan Kozlov","authorDescription":"Ivan is the senior team lead of the AUTO1 Group Kiev engineering office.","authorAvatar":{"relativePath":"pages/auto1-devoxx/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/auto1-devoxx/header.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAJABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAIDBf/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAe21ELH/xAAZEAACAwEAAAAAAAAAAAAAAAACEQABEiD/2gAIAQEAAQUC0TZSuP/EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQMBAT8BP//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8BP//EABcQAAMBAAAAAAAAAAAAAAAAAAAgMTL/2gAIAQEABj8CyRf/xAAaEAEAAgMBAAAAAAAAAAAAAAABADERIFEh/9oACAEBAAE/IaFO5g3PlzMEdP/aAAwDAQACAAMAAAAQAA//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAEDAQE/ED//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/ED//xAAbEAADAAIDAAAAAAAAAAAAAAABESEAMSBBYf/aAAgBAQABPxClqP0uMAdzbvAsCMJhXD//2Q==","width":640,"height":284,"src":"/static/cae90168dbe3b194c3406bc103d1a5f3/c8462/header.jpg","srcSet":"/static/cae90168dbe3b194c3406bc103d1a5f3/c8462/header.jpg 1x"}}}},"html":"“WITH queries”, are SQL constructs giving a possibility of creating temporal data views for a sake of a query execution.\nWITH regional_sales AS (\n SELECT region, SUM(amount) AS total_sales\n FROM orders\n GROUP BY region\n), top_regions AS (\n SELECT region\n FROM regional_sales\n WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)\n)\nSELECT region,\n product,\n SUM(quantity) AS product_units,\n SUM(amount) AS product_sales\nFROM orders\nWHERE region IN (SELECT region FROM top_regions)\nGROUP BY region, product;The beauty of database views
\nAn ugly brother
\nGame changer
\nAS MATERIALIZED the engine would execute it immediately.\nIt is also possible to hint the optimizer that we definitely want the CTE to be inlined, e.g. when the CTE is being referenced twice the engine won't inline it by default.“WITH queries”.JPA or JDBC?
\nDemo time
\n\n@GetMapping(path = \"/v1/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic Order getEntity(@PathVariable(\"order_id\") Integer orderId) {\n return orderRepository.findById(orderId).orElseThrow();\n}orders, order_details, products, customers, customer_customer_demo and customer_demographics.\nYou could check the corresponding JPA mapping here.\n{\n \"orderId\": 10284,\n \"employeeId\": 4,\n \"orderDate\": \"1996-08-19\",\n \"requiredDate\": \"1996-09-16\",\n \"shippedDate\": \"1996-08-27\",\n \"shipVia\": 1,\n \"freight\": 76.56,\n \"shipName\": \"Lehmanns Marktstand\",\n \"shipAddress\": \"Magazinweg 7\",\n \"shipCity\": \"Frankfurt a.M.\",\n \"shipRegion\": null,\n \"shipPostalCode\": \"60528\",\n \"shipCountry\": \"Germany\",\n \"orderDetails\": [\n {\n \"orderId\": 10284,\n \"productId\": 27,\n \"product\": {\n \"productId\": 27,\n \"categoryId\": 3,\n \"supplierId\": 11,\n \"productName\": \"Schoggi Schokolade\",\n \"discontinued\": false,\n \"quantityPerUnit\": \"100 - 100 g pieces\",\n \"unitPrice\": 43.9,\n \"unitsInStock\": 49,\n \"unitsOnOrder\": 0,\n \"reorderLevel\": 30\n },\n \"unitPrice\": 35.1,\n \"quantity\": 15,\n \"discount\": 0.25\n },\n {...},\n {...},\n {...}\n ],\n \"customer\": {\n \"customerId\": \"LEHMS\",\n \"companyName\": \"Lehmanns Marktstand\",\n \"contactName\": \"Renate Messner\",\n \"contactTitle\": \"Sales Representative\",\n \"address\": \"Magazinweg 7\",\n \"city\": \"Frankfurt a.M.\",\n \"region\": null,\n \"postalCode\": \"60528\",\n \"country\": \"Germany\",\n \"phone\": \"069-0245984\",\n \"fax\": \"069-0245874\",\n \"customerDesc\": [\n {\n \"customerTypeId\": \"AAAA\",\n \"customerDesc\": \"First demographic\"\n },\n {...},\n {...},\n {...}\n ]\n }\n}JDBC to the rescue
\n\nWITH products(\"productId\", \"categoryId\", \"supplierId\", \"productName\", \"discontinued\", \"quantityPerUnit\",\n \"unitPrice\", \"unitsInStock\", \"unitsOnOrder\", \"reorderLevel\") AS\n (SELECT p.product_id,\n p.category_id,\n p.supplier_id,\n p.product_name,\n p.discontinued::boolean,\n p.quantity_per_unit,\n p.unit_price,\n p.units_in_stock,\n p.units_on_order,\n p.reorder_level\n FROM products p),\n orderDetails(\"orderId\", \"productId\", \"product\", \"unitPrice\", \"quantity\", \"discount\") AS\n (SELECT d.order_id,\n d.product_id,\n (SELECT to_json(p) FROM products p WHERE p.\"productId\" = d.product_id),\n d.unit_price,\n d.quantity,\n d.discount\n FROM order_details d),\n\n demographics(\"customerId\", \"customerTypeId\", \"customerDesc\") AS\n (SELECT ccd.customer_id,\n ccd.customer_type_id,\n cd.customer_desc\n FROM customer_demographics cd\n JOIN customer_customer_demo ccd on cd.customer_type_id = ccd.customer_type_id),\n customers(\"customerId\", \"companyName\", \"contactName\", \"contactTitle\", \"address\", \"city\", \"region\", \"postalCode\",\n \"country\", \"phone\", \"fax\", \"customerDesc\") AS\n (SELECT c.customer_id,\n c.company_name,\n c.contact_name,\n c.contact_title,\n c.address,\n c.city,\n c.region,\n c.postal_code,\n c.country,\n c.phone,\n c.fax,\n (SELECT to_json(array_agg(to_json(d))) FROM (SELECT \"customerTypeId\", \"customerDesc\" FROM demographics WHERE demographics.\"customerId\" = c.customer_id) d)\n FROM customers c),\n\n orders(\"orderId\", \"employeeId\", \"orderDate\", \"requiredDate\", \"shippedDate\", \"shipVia\", \"freight\",\n \"shipName\", \"shipAddress\", \"shipCity\", \"shipRegion\", \"shipPostalCode\", \"shipCountry\", \"orderDetails\",\n \"customer\") AS\n (SELECT o.order_id,\n o.employee_id,\n o.order_date,\n o.required_date,\n o.shipped_date,\n o.ship_via,\n o.freight,\n o.ship_name,\n o.ship_address,\n o.ship_city,\n o.ship_region,\n o.ship_postal_code,\n o.ship_country,\n (SELECT to_json(array_agg(to_json(d))) FROM orderDetails d WHERE d.\"orderId\" = o.order_id),\n (SELECT to_json(c) FROM customers c WHERE c.\"customerId\" = o.customer_id)\n FROM orders o)\n\nSELECT to_json(orders) AS json_body FROM orders WHERE orders.\"orderId\" = ?to_json function is enough to deal with JSON objects and arrays.orderDetails or the customer into order).\nThe simple idiom (accordingly to_json(...) or to_json(array_agg(to_json(...)))) does the job.customer_customer_demo has been mitigated in the end result, just like the JPA version does by using @ManyToMany.
\n\n
\n
\norder_id becomes orderId).\n@SneakyThrows\n@GetMapping(path = \"/v2/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic String getRawBodyFromDatabase(\n @PathVariable(\"order_id\") Integer orderId) {\n return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n rs.next();\n return rs.getString(1);\n }, orderId);\n}JdbcTemplate which has a negligible overhead and it simplifies our code.PreparedStatement directly) - without the use of Spring.\nEven then the necessity of extracting only a single field from the result set makes the solution less coarse than using JDBC in a usual way\n(in our case it would require managing at least three result sets).\nAll the complexity has been moved down into the query.But I need a POJO!
\nOrder.\nOf course the object won’t be tracked by the JPA whatsoever but we don’t need these capabilities - remember: we are serving persisted data and not mutating them.\n@SneakyThrows\n@GetMapping(path = \"/v3/orders/{order_id}\",\n produces = APPLICATION_JSON_VALUE)\npublic Order getEntityNotUsingJpa(@PathVariable(\"order_id\") Integer orderId) {\n return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n rs.next();\n return unmarshall(rs.getString(1), Order.class);\n }, orderId);\n}\n\n@SneakyThrows\nprivate <T> T unmarshall(String json_body, Class<T> clazz) {\n return objectMapper.readValue(json_body, clazz);\n}Measure, measure, measure
\n\n
\n
\n\n
\n
\nPostgres 12 exclusive
\nSummary
\nLet’s start with fragment…
\n\nprivate val fragmentViewModel: EmailLoginFragmentViewModel by viewModel()\nclass KoinTestApp: Application() {\n override fun onCreate() {\n super.onCreate()\n startKoin {\n androidLogger()\n androidContext(this@KoinTestApp)\n modules(emptyList())\n }\n }\n\n internal fun injectModule(module: Module) {\n loadKoinModules(module)\n }\n}onCreate() method, and we pass emptyList() of modules. We also have a method injectModule() which accepts our test module. Since we already started our application, we are using method loadKoinModules() to add this test module to the graph.How to run custom application
\nAndroidManifest inside androidTest folder, because manifest merger will ignore it if we set our test application inside Manifest like this: <application name=\".KoinTestApp\">. So we could use a custom test runner for that purpose. AndroidJUnitRunner class has a newApplication() method. And we can pass our test application name there, making our test runner to start our test application instead of a real one.\nclass KoinTestRunner: AndroidJUnitRunner() {\n override fun newApplication(\n cl: ClassLoader?, \n className: String?, \n context: Context?): Application {\n return super.newApplication(cl, KoinTestApp::class.java.name, context)\n }\n}build.gradle inside our app module:\nandroid {\n defaultConfig {\n …\n …\n testInstrumentationRunner “path.to.your.package.KoinTestRunner”\n }\n}\nandroid {\n testOptions {\n animationsDisabled = true\n }\n}Test rule for fragments
\n\nabstract class FragmentTestRule<F: Fragment>: \n ActivityTestRule<FragmentActivity>(FragmentActivity::class.java, true) {\n \n override fun afterActivityLaunched() {\n super.afterActivityLaunched()\n activity.runOnUiThread {\n val fm = activity.supportFragmentManager\n val transaction = fm.beginTransaction()\n transaction.replace(android.R.id.content, createFragment())\n .commit()\n }\n }\n \n override fun beforeActivityLaunched() {\n super.beforeActivityLaunched()\n val application = InstrumentationRegistry.getInstrumentation()\n .targetContext.applicationContext as KoinTestApp\n application.injectModule(getModule())\n }\n\n protected abstract fun createFragment(): F\n \n protected abstract fun getModule(): Module\n \n fun launch() {\n launchActivity(Intent())\n }\n}\n\nfun <F: Fragment> createRule(fragment: F, module: Module): FragmentTestRule<F> = \n object: FragmentTestRule<F>() {\n override fun createFragment(): F = fragment\n\n override fun getModule(): Module = module\n }FragmentActivity to host fragment under test. So we need to mention this activity in the manifest. For that purpose I created instance of AndroidManifest inside debug variant in my app module:\n<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<manifest\n xmlns:android=\"http://schemas.android.com/apk/res/android\"\n package=\"com.github.olegosipenko.kointestsample\">\n <application>\n <activity\n android:theme=\"@style/AppTheme\"\n android:name=\"androidx.fragment.app.FragmentActivity\"/>\n </application>\n</manifest>Test itself
\n\n@RunWith(AndroidJUnit4ClassRunner::class)\nclass EmailLoginFragmentTest {\n private val fragmentViewModel: EmailLoginFragmentViewModel = mockk(relaxed = true)\n private val fragment = EmailLoginFragment()\n\n @get:Rule\n val fragmentRule = createRule(fragment, module {\n single(override = true) {\n fragmentViewModel\n }\n })\n\n @Test\n fun testBasicInvocation() {\n onScreen<EmailLoginForm> {\n emailField { typeText(EMAIL) }\n passwordField { typeText(PASSWORD) }\n loginButton.click()\n verify {\n fragmentViewModel.loginWithCredentials(EMAIL, PASSWORD)\n }\n }\n }\n\n class EmailLoginForm: Screen<EmailLoginForm>() {\n val emailField = KEditText { withId(R.id.textFieldEmail) }\n val passwordField = KEditText { withId(R.id.textFieldPassword) }\n val loginButton = KButton { withId(R.id.buttonLogin) }\n }\n}\n\nprivate const val EMAIL = \"some@email.com\"\nprivate const val PASSWORD = \"password\"What the heck is a subform anyway?
\n\n
\nHow Formik can help
\nAuto-connected Field component
\nYup validation schema
\nEnough talking, show me the code
\n\n
\n\n
\nNewUserSubForm directory and our fieldsNames file. It’s just a file that exports constants, don’t worry. It will look like this:\n// UserSubform/fieldsNames.js\nexport const FIRST_NAME = 'firstName'\nexport const LAST_NAME = 'lastName'\nexport const EMAIL = 'email'\nexport const PASSWORD = 'password'Defining the validation schema
\n\n// UserSubform/validationSchema.js\nimport * as yup from 'yup'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nconst MIN_PASSWORD_LENGTH = 8\n\nconst REQUIRED_MESSAGE = 'This field is required'\nconst INVALID_EMAIL_FORMAT = 'Invalid email format'\nconst PASSWORD_TOO_SHOWRT = `The password must be at least ${MIN_PASSWORD_LENGTH} characters long`\n\nexport default yup.object({\n [FIRST_NAME]: yup.string().required(REQUIRED_MESSAGE),\n [LAST_NAME]: yup.string().required(REQUIRED_MESSAGE),\n [EMAIL]: yup\n .string()\n .required(REQUIRED_MESSAGE)\n .email(INVALID_EMAIL_FORMAT),\n [PASSWORD]: yup\n .string()\n .required(REQUIRED_MESSAGE)\n .min(MIN_PASSWORD_LENGTH, PASSWORD_TOO_SHOWRT),\n})\n// UserSubform/defaultValues.js\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nexport default {\n [FIRST_NAME]: '',\n [LAST_NAME]: '',\n [EMAIL]: '',\n [PASSWORD]: '',\n}Rendering the subform
\n\n// UserSubform/index.js\nimport React, { Fragment } from 'react'\nimport { Field, ErrorMessage } from 'formik'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nexport default class NewUserSubForm extends React.Component {\n render() {\n return (\n <Fragment>\n <Field component=\"input\" name={FIRST_NAME} />\n <ErrorMessage name={FIRST_NAME} />\n <Field component=\"input\" name={LAST_NAME} />\n <ErrorMessage name={LAST_NAME} />\n <Field component=\"input\" name={EMAIL} />\n <ErrorMessage name={EMAIL} />\n <Field component=\"input\" name={PASSWORD} />\n <ErrorMessage name={PASSWORD} />\n </Fragment>\n )\n }\n}name property, but the cool thing is that it uses it like the Lodash set function. It means that we can write the name of a field like this: user.firstName. In this way Formik will create an object in its state called user, and then a property inside of user called firstName that will contain the value of our field.Making a subform flexible
\nnamespace. If received, the component will prepend the namespace to every field name. In this way it will be easier to wrap all the subform’s values under a certain object in the main form.\n// UserSubform/index.js\nimport React, { Fragment } from 'react'\nimport PropTypes from 'prop-types'\nimport { Field, ErrorMessage } from 'formik'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nexport default class NewUserSubForm extends React.Component {\n static propTypes = {\n namespace: PropTypes.string,\n }\n\n withNamespace(fieldName) {\n const { namespace } = this.props\n\n return namespace ? `${namespace}.${fieldName}` : fieldName\n }\n\n render() {\n const { withNamespace } = this\n\n return (\n <Fragment>\n <Field component=\"input\" name={withNamespace(FIRST_NAME)} />\n <ErrorMessage name={withNamespace(FIRST_NAME)} />\n <Field component=\"input\" name={withNamespace(LAST_NAME)} />\n <ErrorMessage name={withNamespace(FIRST_NAME)} />\n <Field component=\"input\" name={withNamespace(EMAIL)} />\n <ErrorMessage name={withNamespace(FIRST_NAME)} />\n <Field component=\"input\" name={withNamespace(PASSWORD)} />\n <ErrorMessage name={withNamespace(FIRST_NAME)} />\n </Fragment>\n )\n }\n}The main form
\n\n
\nregistrationForm and it will have the same parts of a subform:\n// CreateNewUserRoute/form/fieldsNames.js\nexport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './subforms/NewUserSubForm/fieldsNames'\nexport const USER = 'user'\n// CreateNewUserRoute/form/validationSchema.js\nimport * as yup from 'yup'\nimport { USER } from './fieldsNames'\nimport userValidationSchema from './subforms/NewUserSubForm/validationSchema'\n\nexport default yup.object({\n [USER]: userValidationSchema,\n})\n// CreateNewUserRoute/form/defaultValues.js\nimport { USER } from './field Names'\nimport userDefaultValues from './subforms/NewUserSubForm/defaultValues'\n\nexport default {\n [USER]: userDefaultValues,\n}\n// CreateNewUserRoute/form/index.js\nimport React from 'react'\nimport { Formik, Form } from 'formik'\nimport NewUserSubForm from './subforms/NewUserSubForm'\nimport validationSchema from './validationSchema'\nimport defaultValues from './defaultValues'\nimport { USER } from './fieldsNames'\nimport ErrorBanner from 'path/to/components/ErrorBanner'\n\nexport default class NewUserSubForm extends React.Component {\n state = {\n unknownErrors: null,\n }\n\n onSubmit = async (values, { setSubmitting, setErrors }) => {\n try {\n // Send values somehow\n await sendForm(values)\n } catch (e) {\n // Map and show the errors in your form\n const [formErrors, unknownErrors] = mapErrorsFromRequest(e)\n\n setErrors(formErrors)\n this.setState({\n unknownErrors,\n })\n } finally {\n setSubmitting(false)\n }\n }\n\n render() {\n const { unknownErrors } = this.state\n\n return (\n <Formik onSubmit={this.onSubmit} initialValues={defaultValues} validationSchema={validationSchema}>\n {() => (\n <Form>\n {unknownErrors && <ErrorBanner errors={unknownErrors} />}\n <NewUserSubForm namespace={USER} />\n </Form>\n )}\n </Formik>\n )\n }\n}Helpful advices on creating subforms
\nA subform should have only first level values in its state
\nKeep the validation schema extensible and scalable
\n\n// UserSubform/validationSchema.js\nimport * as yup from 'yup'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nconst MIN_PASSWORD_LENGTH = 8\n\nconst REQUIRED_MESSAGE = 'This field is required'\nconst INVALID_EMAIL_FORMAT = 'Invalid email format'\nconst PASSWORD_TOO_SHOWRT = `The password must be long at least ${MIN_PASSWORD_LENGTH} characters`\n\nexport default function validationSchema(mandatoryFields = { [LAST_NAME]: true }) {\n return yup.object({\n [FIRST_NAME]: yup.string().required(REQUIRED_MESSAGE),\n [LAST_NAME]: yup.lazy(() => (mandatoryFields.lastName ? yup.string().required(REQUIRED_MESSAGE) : yup.string())),\n [EMAIL]: yup\n .string()\n .required(REQUIRED_MESSAGE)\n .email(INVALID_EMAIL_FORMAT),\n [PASSWORD]: yup\n .string()\n .required(REQUIRED_MESSAGE)\n .min(MIN_PASSWORD_LENGTH, PASSWORD_TOO_SHOWRT),\n })\n}Conclusions
\ntl;dr
\n\n
\nWhat is serverless ?
\nHere comes Bref.
\n\n
\ncomposer require mnapoli/bref\nThen from your project root command line run vendor/bin/bref init\nIt will create two additional files in your project template.yaml and index.php.\nYou can remove index.php for this moment, as it will be unnecessary./tmp dir\nSo we have to extend src/Kernel.php of Symfony with these two functions\n public function getLogDir(): string\n {\n // When on the lambda only /tmp is writeable\n if (getenv('LAMBDA_TASK_ROOT') !== false) {\n return '/tmp/log/';\n }\n\n return $this->getProjectDir().'/var/log';\n }\n\n public function getCacheDir(): string\n {\n // When on the lambda only /tmp is writeable\n if (getenv('LAMBDA_TASK_ROOT') !== false) {\n return '/tmp/cache/'.$this->environment;\n }\n\n return $this->getProjectDir().'/var/cache/'.$this->environment;\n }\nResources:\n Website:\n Type: AWS::Serverless::Function\n Properties:\n FunctionName: 'symfony-website'\n CodeUri: .\n Handler: public/index.php\n Timeout: 30 # in seconds (API Gateway has a timeout of 30 seconds)\n MemorySize: 1024\n Runtime: provided\n Layers:\n - 'arn:aws:lambda:us-east-1:209497400698:layer:php-73-fpm:1'\n Events:\n HttpRoot:\n Type: Api\n Properties:\n Path: /\n Method: ANY\n HttpSubPaths:\n Type: Api\n Properties:\n Path: /{proxy+}\n Method: ANY\n\n Console:\n Type: AWS::Serverless::Function\n Properties:\n FunctionName: 'symfony-console'\n CodeUri: .\n Handler: bin/console\n Timeout: 120 # in seconds\n Runtime: provided\n Layers:\n - 'arn:aws:lambda:us-east-1:209497400698:layer:php-73:1' # PHP\n - 'arn:aws:lambda:us-east-1:209497400698:layer:console:1' # The \"console\" layerFunctionName key.\nTheir execution will be available thanks to API Gateway which also will be created during the deploy process.\nFor testing purpose, you don't have to change template file much. Please remember to check your AWS default region if it's different than us-east-1 just change it in all Layers key of your Resources.aws s3 mb s3://<bucket-name>\nThen upload the code and generate the stack configuration:\nsam package \\\n --output-template-file .stack.yaml \\\n --s3-bucket <bucket-name>\nsam deploy \\\n --template-file .stack.yaml \\\n --capabilities CAPABILITY_IAM \\\n --stack-name <stack-name>\n
\n\n
\nFurther info
\n\n
","fields":{"slug":"/php-serverless-with-bref/","tags":["Symfony","PHP","serverless"]}}},{"node":{"id":"92132641-afb7-53d7-8663-09c7b3dec866","frontmatter":{"category":"DevOps","title":"Monitoring screens on Raspberry Pi","date":"2019-05-02","summary":"How to setup monitoring tools on TV screens using Raspberry Pi","thumbnail":{"relativePath":"pages/monitoring-screens/screens.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAALABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAMEAQX/xAAXAQADAQAAAAAAAAAAAAAAAAABAgME/9oADAMBAAIQAxAAAAGuV+ZqLOeBv//EABoQAQADAAMAAAAAAAAAAAAAAAEAAhEDEjP/2gAIAQEAAQUCyyd2NwZhOb0//8QAFxEAAwEAAAAAAAAAAAAAAAAAAAIRUf/aAAgBAwEBPwGNpT//xAAYEQADAQEAAAAAAAAAAAAAAAAAAhEBMf/aAAgBAgEBPwGpOGKf/8QAGRAAAgMBAAAAAAAAAAAAAAAAAAEQITER/9oACAEBAAY/Au0aW4wZ/8QAGBAAAwEBAAAAAAAAAAAAAAAAAAERMSH/2gAIAQEAAT8hUE0UuHcTUWQVsCF//9oADAMBAAIAAwAAABBMD//EABcRAQEBAQAAAAAAAAAAAAAAAAEAESH/2gAIAQMBAT8Q7axV/8QAGBEBAQADAAAAAAAAAAAAAAAAAQARIaH/2gAIAQIBAT8QC07hTN//xAAaEAEBAQEAAwAAAAAAAAAAAAABEQBRMUGx/9oACAEBAAE/EOs8EuY6EsQM8KHpwGjZyuCJ5OYFE583/9k=","width":578,"height":325,"src":"/static/adf3f762bfd697b8fdc141491acaadb2/a2998/screens.jpg","srcSet":"/static/adf3f762bfd697b8fdc141491acaadb2/a2998/screens.jpg 1x,\n/static/adf3f762bfd697b8fdc141491acaadb2/99cce/screens.jpg 1.5x,\n/static/adf3f762bfd697b8fdc141491acaadb2/6e995/screens.jpg 2x,\n/static/adf3f762bfd697b8fdc141491acaadb2/968b1/screens.jpg 3x"}}},"authorName":"Konstantin Shcherban","authorDescription":"Konstantin is DevOps Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/monitoring-screens/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"\nFROM resin/%%BALENA_MACHINE_NAME%%-alpine:3.7\n\nRUN apk add --update --no-cache \\\n xf86-video-fbdev \\\n xf86-input-libinput \\\n mesa \\\n mesa-dri-swrast \\\n xorg-server \\\n mesa-egl \\\n xf86-input-evdev \\\n xdotool \\\n bash \\\n firefox-esr\n\nADD profiles.ini /root/.mozilla/firefox/profiles.ini\nADD run /bin/run\n\n# set timezone\nARG TIMEZONE=Europe/Berlin\nRUN apk add --update --no-cache tzdata && \\\n cp /usr/share/zoneinfo/$TIMEZONE /etc/localtime && \\\n echo $TIMEZONE > /etc/timezone && \\\n apk del tzdata\n\n## comment out if you don't want systemd\nENV INITSYSTEM on\nENV DBUS_SYSTEM_BUS_ADDRESS unix:path=/host/run/dbus/system_bus_socket\n\nENTRYPOINT [ "/bin/run" ]\n[General]\nStartWithLastProfile=1\n\n[Profile0]\nName=default\nIsRelative=0\nPath=/data/firefox-profile\nDefault=1\n
\n\n#!/bin/bash\n# URL is set in balena device settings\n# it should point to dashkiosk\nURL=${URL:-http://jodel-app.com}\n\n# By default docker gives us 64MB of shared memory size but to display heavy\n# pages we need more.\numount /dev/shm && mount -t tmpfs shm /dev/shm\n\nrm /tmp/.X0-lock &>/dev/null || true\n\nmkdir -p /data/firefox-profile\nstartx /usr/bin/firefox \"$URL\" -- -dpms -s off -s noblank -nocursor&\n# wait until Xserver and Firefox start\nsleep 40\nexport DISPLAY=:0\nxdotool search --name \".*Mozilla Firefox\" windowsize 100% 100%\nxdotool search --name \".*Mozilla Firefox\" key \"F11\"\n# sleep forever\ntail -f /dev/nullRESIN_HOST_CONFIG_gpu_mem=128 sets 128MB of GPU memory, by default it's only 16MB.RESIN_HOST_CONFIG_dtoverlay=\"vc4-fkms-v3d\" enables new 3d driver.MVC approach
\nFull Stack development.hard to build for a single person. Knowing that development cycle of such complex pages could be split between several software engineers, we divided responsibilities between back- and front-end developers. We decided to generate the View part of MVC architecture in client's browser instead of application's server.SPA paradigm
\nrequests each time full page content (including static content, such as layout, menus, etc), or it requests only the important (dynamic) content. This difference can be seen in a diagram below.SPA evolution in AUTO1
\nFrom Monolith to php-rendered SPA
\n\n
\nUnder the hood of SPA controller we implemented such functionality:\n
\n\n
\n\n<!doctype html>\n<html>\n <head>\n <meta charset=\"utf-8\"><meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n <meta name=\"viewport\" content=\"width=device-width,initial-scale=1,shrink-to-fit=no\">\n <title>SPA index</title>\n <link href=\"https://cdn.xyz.de/1.2.3/main.css\" rel=\"stylesheet\">\n </head>\n <body>\n <noscript>You need to enable JavaScript to run this app.</noscript>\n <div id=\"app\"></div>\n <script type=\"text/javascript\" src=\"https://cdn.xyz.de/1.2.3/main.js\"></script>\n </body>\n</html>AUTO1 Group
\n#spa #monolith #mvc #engineering
","fields":{"slug":"/spa-evolution/","tags":["auto1","SPA","monolith","MVC"]}}},{"node":{"id":"4847d312-706f-5678-9e9e-1a333f42095e","frontmatter":{"category":"Coding","title":"How we use ES6 generators to simplify our React Redux application code","date":"2019-04-04","summary":"How we use ES6 generators instead of thunk to simplify our React Redux application code and improve productivity","thumbnail":{"relativePath":"pages/es6-generators-react-redux/thumbnail.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAnLAAAJywEjDYgKAAADmUlEQVQ4y3VV2W/MURQemUY7+7Szd2Y681umLSHWB4TW1nqo5YUisYVIVGsJEXtrryUIam9Va0nQ1hoULYrEFhL8ASS2+A94+Zxzfp0xpR5ucnPPud893znfOdekqioUWqqiyFI0DVooAN2bDTUagaLrYpdFez5jm5YbFF+5k8SgZUoHVFQCi4QRmlIOb8V65A0fAd2Xk7rA++iIMfAsq0Zu2XRo4dweoClAOaDXtYAPgTlLkNn6Cq5tx5HV+hLu6sMUVRRqLM84a3kB95Y6ZLa9RnDGAmhBP4HqBsP0CIUq0bCdvInQtNnIt5gRGzgIjgMXYD/SAvvRNjj3NSPefwAKMvsgMHuxnHMQDJiKUPmLsvVMOyKjxxtUiL4WCsL89BsyHn+WaNS8qPiHJ5bJ42o89ifHEmEyqXpCAOxHWxEZW4qEwyIAjsOXkb1uL7I37Ifj4EWJKGHPQrhkChyHLlEeQ1AS+amgTCm65Kj7veJkO3UL2RsPwNr0AKYPv2Bt7oDlXKexb7wn4Lb623DsPw/d4+7Oo9YdYTwu1fItXi0AfW++Q+a1t/Cs2i5Uo6PHIV7QD0pBISLFpTA/+w7Pyq3ix8t69j7886sk/xIh64oTb23qoAsliA0eRrm5IZF6K9ZRdWNw7m2Cc/cZ0SBLhqnbTtxAbNBQhCeUwXKxS+wsOZNv0SoyXpfQOeGsNeeeRmQ8/ER7j9Ay/YAsZ22DpMX85AtcO06SwHMMoZMfp8A/r5IAF66U1wSQohFAei2j86OxJ6kkAV27TgtgRtdn2teLXYLwM+Ad+OcuTaNMuYiOKibtDRY5sMa8S9bKBb7MEWmRXIMy+QvlIcNJEZNgucCUG6QWqaLwZcv5R1SQN9IF3sqNMHd9kUdEFrQiYyZQob7CW7VJ/Pre+iCF9C9Y/qcohiAN2bCI7ceukrjvwl1TJw+Y3v+EpblTpJPcuzcfEmkxC06BFvSlySZN2Fwle12rVDvhssnUsR+5IsLOWVNLthY5SzitiBRNlHY0JlLiL2EnW49esTa2i/ak9ciZRcsVZ6rcFQJANga0NtxNa9tk6/UYDiGiQsNh6izkZ5kRL+wH187TUiSWFlc/Xtgf+TQceNJwetK7pNfx5Z9TIeOLx1bWledw1dYbo4kk5dp5Ss7YxoULzFxk5E/vZXz1GLAUIcsjOrLI0Fo3JRmwVHXPihqEJpf/f8D+9wsg8H++ADqTgRDq/Qv4DWMLuHUxNoXrAAAAAElFTkSuQmCC","width":325,"height":325,"src":"/static/2bee11a830bacc5ae9006df56b20c33a/b3029/thumbnail.png","srcSet":"/static/2bee11a830bacc5ae9006df56b20c33a/b3029/thumbnail.png 1x,\n/static/2bee11a830bacc5ae9006df56b20c33a/8d141/thumbnail.png 1.5x,\n/static/2bee11a830bacc5ae9006df56b20c33a/ee72c/thumbnail.png 2x,\n/static/2bee11a830bacc5ae9006df56b20c33a/5dfa8/thumbnail.png 3x"}}},"authorName":"Chirag Swadia","authorDescription":"Chirag is a Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/es6-generators-react-redux/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAECAwT/xAAXAQEBAQEAAAAAAAAAAAAAAAAEAwAB/9oADAMBAAIQAxAAAAGIveV5aObk0FjsCs//xAAbEAADAAIDAAAAAAAAAAAAAAAAAQIRIRIiQf/aAAgBAQABBQKnxmHTMGhGz0l9f//EABoRAAICAwAAAAAAAAAAAAAAAAABAhAxMkH/2gAIAQMBAT8B7U9mLB//xAAZEQABBQAAAAAAAAAAAAAAAAAAAQIQMTL/2gAIAQIBAT8BhuRbP//EABsQAAIBBQAAAAAAAAAAAAAAAAARAgEQIDFB/9oACAEBAAY/AmKcEatzCh//xAAbEAEAAwADAQAAAAAAAAAAAAABABEhMVFh8P/aAAgBAQABPyEnrfkRsPknmGxTsy8Dqp82W0JRuRFFn//aAAwDAQACAAMAAAAQ+Dj/AP/EABgRAAMBAQAAAAAAAAAAAAAAAAABETEQ/9oACAEDAQE/EG5ovBkf/8QAGBEAAgMAAAAAAAAAAAAAAAAAADEQESH/2gAIAQIBAT8Q2lCBh//EABwQAQADAQEAAwAAAAAAAAAAAAEAESExUXGB8P/aAAgBAQABPxCyt4AerkU0XY8TkPRXxBSQfql1NpWO/co8qKgOPZRgcYmBU9n/2Q==","width":50,"height":50,"src":"/static/8cf5c37636309895b29dd7ce5f1fcf28/d2d31/avatar.jpg","srcSet":"/static/8cf5c37636309895b29dd7ce5f1fcf28/d2d31/avatar.jpg 1x,\n/static/8cf5c37636309895b29dd7ce5f1fcf28/0b804/avatar.jpg 1.5x,\n/static/8cf5c37636309895b29dd7ce5f1fcf28/753c3/avatar.jpg 2x,\n/static/8cf5c37636309895b29dd7ce5f1fcf28/31ca8/avatar.jpg 3x"}}},"headerImage":{"relativePath":"pages/es6-generators-react-redux/headerImage.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsSAAALEgHS3X78AAABWUlEQVQY022QS0/CQBSFOzMtFLCdtsOrPOQRIPJogIILgkY0GqKJG5cu3GqMxIULExOJG/9s/0N7nBbjRiaZnJs7ky/nHEWRx63WRDqVCjRdR344DUV/jPxohlQmC0YpNE2Dqqp/SqkC+8BFuzBHTucRYzSK3xzHOY95SrXXF1KCbK2Fzv1T6L1+ofewQa7egtz/u0Sh6BZO0HBmOLT9SO4iohAYhrED6ooidJMHldUNvJdt6G22Uj/hXtyiO/TgTyeY+j4WiwUKpTxEtomheyVhE4wq64jr5RgKi/86LJZdQQkJ7MEUg8f3cPz2jeHzB+zRHLIKcM5hmqZUE2ktg4bto1NYomZ56BVPo6Yzjwgh8Z8d0G22BZGRWTaH8vIyrK/vUDm7hpoz9kauWqMk8lFphZY4jspGL3HIubUD5oUQjKlB0o+qhokythf21yOhiVLCkg7jOXb4A0fehWKiiDXIAAAAAElFTkSuQmCC","width":757,"height":236,"src":"/static/c49511e796013b6394df522c11a4a9fc/7eaf7/headerImage.png","srcSet":"/static/c49511e796013b6394df522c11a4a9fc/7eaf7/headerImage.png 1x"}}}},"html":"Background
\n\nconst INCREMENT_COUNTER = 'INCREMENT_COUNTER';\n\nconst increment = () => ({\n type: INCREMENT_COUNTER\n})\nconst incrementAsync = () => {\n return dispatch => {\n setTimeout(() => {\n dispatch(increment());\n }, 1000);\n };\n}The Problem
\n\nconst makeSandwichesForEverybody = () => (dispatch, getState) => {\n if (!getState().sandwiches.isShopOpen) {\n return Promise.resolve();\n }\n\n return dispatch(\n makeASandwichWithSecretSauce('My Grandma')\n ).then(() =>\n Promise.all([\n dispatch(makeASandwichWithSecretSauce('Me')),\n dispatch(makeASandwichWithSecretSauce('My wife'))\n ])\n ).then(() =>\n dispatch(makeASandwichWithSecretSauce('Our kids'))\n ).then(() =>\n dispatch(getState().myMoney > 42 ?\n withdrawMoney(42) :\n apologize('Me', 'The Sandwich Shop')\n )\n );\n};ES6 Generators to the rescue
\n\nconst makeSandwichesForEverybody = () => function* ({ getState }) {\n if (!getState().sandwiches.isShopOpen) {\n return;\n }\n\n yield makeASandwichWithSecretSauce('My Grandma');\n yield [makeASandwichWithSecretSauce('Me'), makeASandwichWithSecretSauce('My wife')];\n yield makeASandwichWithSecretSauce('Our kids');\n\n if (getState().myMoney > 42) {\n yield withdrawMoney(42):\n } else {\n yield apologize('Me', 'The Sandwich Shop');\n }\n};Testability
\nError Handling
\n\nconst catchError = error => console.error(error); \nconst middlewares = applyMiddleware(createGeneratorMiddleware(null, catchError));\nconst someAction = () => function* ({ api }) {\n try {\n yield api.loadSomeItems();\n } catch (error) {\n // handle error logic\n console.log(error);\n }\n}\nconst someAction = () => function* ({ api }) {\n yield api.loadSomeItems();\n}catchError function.Enough of Making Sandwiches
\n\nconst loadAds = (filters) =>\n function*() {\n // filter is an object with some information like {make: 'Audi', model: 'A3'}\n \n const { ads: result } = yield api.findAds(filters);\n\n // Once we get the ads list, we just save it in the redux store\n }\n\nconst loadSimilarAds = (filters) =>\n function*(){\n // this is same as loadAds, but the API endpoint from which it fetches data is different\n }loadDataFromCms, trackGtmEvents etc. Now the action which will be called when the main component mounts, will be as shown below\nexport const loadPage = () => \n function*({ getState }){\n yield loadDataFromCms('header');\n\n const filters = yield getFiltersFromStore(getState());\n yield [\n loadAds(filters),\n loadMakeModelSubtype(filters),\n loadDataFromCms('home')\n ];\n\n // If no ads found, we try to find similar ads which might interest the user\n if( getState().ads.length === 0 ) {\n yield loadSimilarAds(filters);\n }\n }Summary
\nTL;DR
\nFull Story
\nevent-store service, one simply can schedule a job to be executed at a specific time in future. There are already several ready to use tools for this specific purpose (e.g. BigBen, Quartz), however, because of several reasons (which are beyond the scope of this post) we decided to implement our own from scratch.Deadlocks, we need a brief introduction on event-store's implementation.events table is as follow:\n\n
\n\n \n\n\nid \nevent_type \ndue_time \nstart_process_time \nend_process_time \n\n \n\n'5c55ea79-4774-49ae-96e2-975af7a41ce6' \n'user-cache-flush' \n'2019-01-01 00:00:00' \nnull \nnull \n
\na. The service periodically fetches a bunch of unprocessed events (those that are due and have no process time, neither start nor end) from the database (and marks them as progressing by filling the start_process_time column with the current timestamp). Let's call this fetch-unprocessed query.end_process_time column with the current timestamp). Let's call this save-processed query.
\na. The service periodically checks the table and looks for stale events (those that are due and have start_process_time but no end_process_time). Let's call this fetch-stale query.start_process_time to null). Let's call this release-stale query.\nERROR: deadlock detected\nDetail: Process 5234 waits for ShareLock on transaction 3465; blocked by process 467845.\n Process 467845 waits for ShareLock on transaction 96575; blocked by process 5234.\nHint: See server log for query details.\nWhere: while updating tuple (14954,4) in relation \"events\"\nERROR: deadlock detected\nDetail: Process 10438 waits for ExclusiveLock on tuple (14954,4) of relation 19118 of database 19113; blocked by process 31501.\n Process 31501 waits for ShareLock on transaction 763124271; blocked by process 28450.\n Process 28450 waits for ShareLock on transaction 763124277; blocked by process 28873.\n Process 28873 waits for ExclusiveLock on tuple (14954,4) of relation 19118 of database 19113; blocked by process 10438.\nHint: See server log for query details.\nWhere: while locking tuple (6984,19) in relation \"events\"\n-- fetch-unprocessed query\nUPDATE events event_to_update\nSET start_process_time = now()\nFROM (\n SELECT id\n FROM events \n WHERE end_process_time is NULL\n AND start_process_time IS NULL\n AND (due_time BETWEEN ? AND ?)\n) matched_event_for_update\nWHERE event_to_update.id = matched_event_for_update.id\nRETURNING *;\n\n-- release-stale query (including fetch-stale query)\nUPDATE events event_to_release\nSET start_process_time = NULL\nFROM (\n SELECT id\n FROM events\n WHERE end_process_time IS NULL AND start_process_time < ?\n) stuck_event\nWHERE event_to_release.id = stuck_event.id\nRETURNING *;Serializable. However this is the last solution we wanted since it would hurt the performance and the quality of the service (think what would happen if one service start a transaction, fetch the events to process, but for some reason become stale).\n
\n\n-- fetch-unprocessed query\nUPDATE events event_to_update\nSET start_process_time = now()\nFROM (\n SELECT id\n FROM events \n WHERE ... -- as before\n ORDER BY ID\n FOR NO KEY UPDATE\n) matched_event_for_update\nWHERE ... -- as before\nRETURNING *;\n\n-- release-stale query (including fetch-stale query)\nUPDATE events event_to_release\nSET start_process_time = NULL\nFROM (\n SELECT id\n FROM events\n WHERE ... -- as before\n ORDER BY ID\n FOR NO KEY UPDATE\n) stuck_event\nWHERE ... -- as before\nRETURNING *;\nERROR: deadlock detected\nDetail: Process 7534 waits for ShareLock on transaction 93897000; blocked by process 13376.\n Process 13376 waits for ShareLock on transaction 93896994; blocked by process 7534.\nHint: See server log for query details.\nWhere: while rechecking updated tuple (134131,3) in relation \"events\"JpaRepository.save(Iterable<S> entities). In the code, after fetching events to be processed, we flag (UPDATE) the process events as processed, and also save (INSERT) new events spawned by the processed events. Here's the catch: the event processing is done in parallel. So the resulting list of events (both processed and newly created ones) is not ordered with respect to other transactions. So the third refactoring was to restore the order of processed events:\n/**\n * To process events.\n * The result list will be ordered with respect to the given list.\n * All new events will be added at the end of the result list.\n * @param events events to be processed\n * @return list of processed and created events\n */\nprivate List<Event> doProcessEvents(List<Event> events) {\n ProcessingResult result = eventProcessor.processInParallel(events);\n\n List<Event> successfulEvents = result.getSucceeded(); // may contain new events;\n List<Event> failedEvents = result.getFailed();\n List<Event> allEvents = ListUtils.union(successfulEvents, failedEvents);\n\n Map<UUID, Integer> eventIndices = range(0, events.size()).boxed()\n .collect(CollectorUtils.toMap(i -> events.get(i).getId(), i -> i));\n\n // This should sort according to the original order and put all new events at the end.\n allEvents.sort(comparingInt(event -> eventIndices.getOrDefault(event.getId(), Integer.MAX_VALUE)));\n\n return allEvents;\n}Problem statement
\n\n
\nSpring features to be avoided
\n
\n@ActiveProfiles and @ContextConfiguration with different attributes\n
\n@ActiveProfiles({\"integration-test\", \"pg-test\", \"mongo-test-two-nodes\"})@ContextConfiguration(classes = {Application.class, AwsTestConfiguration.class})@ActiveProfiles({\"integration-test\", \"pg-test\"})\n@ActiveProfiles({\"pg-test\", \"integration-test\"})
\n@SpyBean and @MockBean@SpyBean annotation with configuration class that wraps real spring bean into Mockito.spy(..):\n@Configuration\npublic class SomeComponentSpyFactory {\n\t@Autowired\n\tprivate SomeComponent someComponent;\n\n\t@Bean\n\t@Primary\n\tpublic SomeComponent someComponentSpy() {\n \t\treturn Mockito.spy(someComponent);\n\t}\n}@Autowired and @Spy annotations.\n\t@InjectMocks\n\t@Autowired\n\tprivate ParentComponent parentComponent;\n\n\t@Spy\n\t@Autowired\n\tprivate ChildComponent childComponent;
\n@DirtiesContextChallenge: test execution order and cleaning collections (caches)
\ngetAll()-like operations. Those tests were running functions that fetched all records from some collections. This was not a problem as embedded mongo DB was different instance. After all tests started to exploit the same spring context some of them started to fail intermittently. The failures were intermittent because test execution order is different. One service may break another, but not vice versa.\nSo, to overcome this, we added missing @Before operations that cleaned those collections.@After operations that cleaned up, but not everything produced by the test was cleaned up. This caused intermittent failures of other tests.Tools we created to reduce test execution time
\nJenkins PR pipeline step
\nIntelliJ IDEA plugin
\n\n
\n\n
\n\n
\n\n
\nOriginal AUTO1 architecture
\n\n
\n\n
\nResults
\n\n
\nThe situation
\nA new hope
\nDemystifying the indexer
\n\ncom.auto1.playground.service.SpringDummyService=org.springframework.stereotype.Component\ncom.auto1.playground.domain.SpringDummy=javax.persistence.Entity,javax.persistence.Table,javax.persistence.EntityListeners\ncom.auto1.playground.repository.SpringDummyRepository=org.springframework.stereotype.Component,org.springframework.data.repository.Repository\ncom.auto1.playground.configuration.LocalConfiguration=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomService=com.auto1.playground.service.IndexedCustomService\ncom.auto1.playground.Application=org.springframework.stereotype.Component\ncom.auto1.playground.SpringDummyController=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomExample=com.auto1.playground.annotation.IndexedCustom\n<build>\n <plugins>\n <plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.5</version>\n <configuration>\n <annotationProcessorPaths>\n <path>\n <groupId>org.springframework</groupId>\n <artifactId>spring-context-indexer</artifactId>\n </path>\n </annotationProcessorPaths>\n </configuration>\n </plugin>\n </plugins>\n</build>\n
\n\n@Indexed\npublic class IndexedCustomService { }\n@Target(ElementType.TYPE)\n@Retention(RetentionPolicy.RUNTIME)\n@Documented\n@Indexed\npublic @interface IndexedCustom { }\n\n@IndexedCustom\npublic class IndexedCustomExample { }\n@SpringBootApplication\n@ComponentScan(basePackages= {\"com.auto1\"}, includeFilters = {\n @ComponentScan.Filter(type= ANNOTATION, value = {IndexedCustomExample.class}),\n @ComponentScan.Filter(type= ASSIGNABLE_TYPE, value = {IndexedCustomService.class})\n }\n)\npublic class Application implements WebMvcConfigurer {\n // ...\n}Test me one more time
\nThe expected disappointment
\nIntro
\n\nSteps:\n- Create item, put item in certain state\n- Create user\n- Create user delivery address\n- Create user billing address\n- Create user payment details\n- Sell item to user\n- User does something with an itemBack in times…
\nRemoving dependencies
\n\n
\n![\"machingun\"](\"/static/machingun-f6e9bbc355090a0e8c6db2e891dfb64a.gif\" "\\"Suitemachingun\\"")
Introducing Q-Service
\n\n
\n\n
\nHow does it work?
\nSummary
\ntl;dr
\nIntro
\nDocker at home
\nLevel 1: Basic usage
\n\ndocker run postgresLevel 2: Next step
\nDocker Compose comes to help. \nversion: '3'\nservices:\n myService1:\n image: service1\n {configuration:}\n myService2:\n image: service2\n {configuration:}docker-compose.yml file ready we are able to start it with one terminal command.\ndocker-compose upLevel 3: Hands-on experience
\nDefine PostgreSQL
\ndocker-compose.yml file and add PostgreSQL as datasource.\nversion: '3'\nservices:\n myDatasource:\n image: postgresmyDatasource that will be created using postgres image.Define Grafana
\n\nversion: '3'\nservices:\n myDatasource:\n image: postgres\n myMonitoring:\n image: grafana/grafana\n ports:\n - 3000:3000myMonitoring that will be created using grafana/grafana image.\nAdditionaly to basic service definition we have also exposed port 3000, so Grafana user interface can be access from outside the container.Troubleshooting
\ndocker-compose up two containers should be configured and run by Docker.\nThen navigating to localhost:3000 will allow us to access Grafana web interface where datasource could be set up.\nThing that is worth notice is that if we would like to configure our PostgreSQL datasource with default suggested Host - localhost:5432Space Exploration and Tourism
\nAnd the best idea of them all
\nFinal Words
\nAnd the really really final words
\n\n
\nReact Widgets: Traditional Approach
\n\n.\n├── server\n └── ....\n├── frontend\n ├── widgets\n ├── login.js\n └── subscribe.js\n └── components\n ├── Login.js\n ├── Subscribe.js\n └── ....\n├── dist #ignored\n ├── widget_login_bundle.js\n └── widget_subscribe_bundle.js\n├── webpack.config.js\n└── package.json/frontend/widgets directory will contain entry points for all the widgets, each file inside should render a react app in a corresponding DOM element id/frontend/widgets/login.js\nimport React from 'react';\nimport ReactDOM from 'react-dom';\nimport Login from '../components/Login';\n\nReactDOM.render(\n <Login />,\n document.getElementById('widget:login'), // <-- NOTE the element id\n);webpack.config.js will look like this\nmodule.exports = {\n entry: [\n './frontend/widgets/login.js',\n './frontend/widgets/subscribe.js',\n ],\n output: {\n filename: 'widget_[name]_bundle.js',\n }\n ...\n}webpack from the project folder will create two bundle files widget_login_bundle.js and widget_subscribe_bundle.js in the dist folder.\n...\n<div id=\"widget:login\"></div>\n<script defer src=\"/dist/widget_login_bundle.js\" />\n...Pros
\n\n
\nCons
\n\n
\n
\nReact Widgets: Standalone Approach
\n\n
\nexample.com/api/widget/save-or-update (Save or upload a widget entry in the database)example.com/api/content-server/upload (Upload an asset file to content server)\n.\n├── frontend # Same as before\n├── dist # Same as before\n└── scripts\n ├── deployWidget.js\n └── ....\n├── webpack.config.js\n└── package.json/scripts/deployWidget.js is a nodejs script that expects a --widget argument and will do the following:\n
\n\nconst path = require('path');\nconst argv = require('yargs').argv;\nconst uploadAssetToContentServer = require('./uploadAssetToContentServer');\nconst createOrUpdateWidgetEntry = require('./createOrUpdateWidgetEntry');\n\nconst deployWidget = async ({ widget }) => {\n const bundleFile = path.join(\n process.cwd(),\n 'dist',\n `widget_${widget}_bundle.js`\n );\n const uploadedUrl = await uploadAssetToContentServer(bundleFile);\n const html = `\n <div id=\"widget:${widget}\"></div>\n <script defer src=\"${uploadedUrl}\"></script>\n `;\n await createOrUpdateWidgetEntry({\n name: widget,\n html,\n });\n console.log(`widget ${widget} deployed successfully`);\n};\n\ndeployWidget({\n widget: argv.widget,\n}).catch(console.error);uploadAssetToContentServer.js and createOrUpdateWidgetEntry.js respectively, which you will need to implement.package.json\n...\n\"scripts\": {\n ....\n \"deploy-widget\": \"webpack && ./scripts/deployWidget\"\n}\nnpm run deploy-widget -- --widget login\n...\n{{ getWidgetHtmlFromDB('login') | raw }}\n...Pros
\n\n
\nCons
\n\n
\n
\nWhat's next
\n\n
","fields":{"slug":"/embeddable-leightweight-react-widgets/","tags":["auto1","engineering","react","widgets","refactoring","modularity"]}}},{"node":{"id":"2b607647-03dd-50d4-8f72-310a92d54e79","frontmatter":{"category":"Coding","title":"Yet another story about a monolith and microservices","date":"2018-12-04","summary":null,"thumbnail":{"relativePath":"pages/vigolante/square-gopher.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAABpklEQVQoz32SS0sCURiG/Qn9gH5BUHtXLWsXbdto3h3NQAMpESpoExIpVJDUpiBwEQTSBa3UURscrzM6F8PLjKMJujBRy0vZqCVi6uHjnMPH+/Ce9+NwACzTKwXOCJG02OnXunxyd0gUpZREtrO7/Me+sAkKST0ROZHt69ni9A4lS6LUAYxUarVGvf6Wz++BPn6U1kPhYrHYZler6SPiIi+iIHMARg/CtAxn1oPklcVybjZ/d7RtNJHkeSKn1jsJn4eTJNtpVCu7ICwmcorYIByjJbHMFprUqNVcLrfLthMUxQY5urevLC/hBMF2PsvvehCWkENwN7AISV24oWqh0P5qtT5qhueXWe3O4uZ2MEm1ms1yqXQJBYSBuGIgNqd/k+OMJEDuQ+GzIKoDYY3ZNDM9tTA/p7u51jhglRMWhBLAyIH9+hNZMc7w8ZzQG4nbjZHbQwY8cVuNgnASeM2zrxsUD8MdHqPXsIwwShsebKjHBYJO1aMLwBjgL+ckuDc/GZHbgHGTzWF4cstZ8p/neLjjn5HG6FUkLUBSIwWT4P7PUY7x7NUPicXZxAqt2owAAAAASUVORK5CYII=","width":570,"height":325,"src":"/static/dc7ba27f8138dca1c3ae8defbfbb334d/b3029/square-gopher.png","srcSet":"/static/dc7ba27f8138dca1c3ae8defbfbb334d/b3029/square-gopher.png 1x"}}},"authorName":"Dmytro Malyshenko","authorDescription":"Team Lead Software Engineer","authorAvatar":null,"headerImage":{"relativePath":"pages/vigolante/golang-banner.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAFABQDASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAAAAIE/8QAFQEBAQAAAAAAAAAAAAAAAAAAAQT/2gAMAwEAAhADEAAAAdNFEwD/AP/EABcQAQADAAAAAAAAAAAAAAAAAAABESH/2gAIAQEAAQUCRi3/xAAVEQEBAAAAAAAAAAAAAAAAAAAAEf/aAAgBAwEBPwFX/8QAFREBAQAAAAAAAAAAAAAAAAAAABH/2gAIAQIBAT8BR//EABQQAQAAAAAAAAAAAAAAAAAAABD/2gAIAQEABj8Cf//EABgQAQADAQAAAAAAAAAAAAAAAAEAESFR/9oACAEBAAE/IYrWRWHMn//aAAwDAQACAAMAAAAQg+//xAAWEQEBAQAAAAAAAAAAAAAAAAABABH/2gAIAQMBAT8QHGVt/8QAFhEBAQEAAAAAAAAAAAAAAAAAAQAR/9oACAECAQE/EE0gZf/EABgQAQEBAQEAAAAAAAAAAAAAAAERACEx/9oACAEBAAE/EM/EsTpfcEVGYUK7/9k=","width":1280,"height":333,"src":"/static/b2168e9dcfd879a80bb3db0d5b429775/966a5/golang-banner.jpg","srcSet":"/static/b2168e9dcfd879a80bb3db0d5b429775/966a5/golang-banner.jpg 1x,\n/static/b2168e9dcfd879a80bb3db0d5b429775/aa36d/golang-banner.jpg 1.5x"}}}},"html":"Standalone Approach\n
\ndeployWidget.js script.deployWidget.js to deploy widgets to QA env. and create the scary-red \"Move to production\" button in QA CMS to just copy the entry from QA DB to Production DB.Once upon a time, there was a startup which was built as a monolith...
\n\n
\nThe moment any monolith is afraid of
\nThere should be a better way
\n\n
\n\n
\n\n
\n\"ElastAlert Trips & Tricks\"
\nHow do buffer_time, the elastalert run model, late indexed events and timeframes play together?
\nWhat is the problem when using use_count_query for frequency rules?
\nIf there is an in memory storage, what happens on Rule Reloading and service restart?
\nHow can I return the documents that caused the alert?
\nWhat Index pattern should I set?
\nHow should a efficient rule look like?
\n\n","fields":{"slug":"/tech-center-in-katowice/","tags":["auto1","engineering","social"]}}},{"node":{"id":"be97e700-7d99-5a92-a5b4-9fb3fbf0c3c3","frontmatter":{"category":"Social","title":"One more Hackathon at AUTO1","date":"2018-06-20","summary":"Event diary of our 2018 Spring Hackathon, UN<->DOCK","thumbnail":{"relativePath":"pages/one-more-hackathon-at-auto1/auto1-hackathon-builders-in-action.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAQEBAQAAAAAAAAAAAAAAAAUCBAP/xAAXAQEBAQEAAAAAAAAAAAAAAAACAQAD/9oADAMBAAIQAxAAAAHppRcDrccBCV5kWRp//8QAGxAAAgIDAQAAAAAAAAAAAAAAAQMAAgQSEzH/2gAIAQEAAQUCabnIr5HMHYtBi3HWw2GxqOt5/8QAGREAAQUAAAAAAAAAAAAAAAAAAgAQERIx/9oACAEDAQE/ARGdVX//xAAXEQEAAwAAAAAAAAAAAAAAAAABABAR/9oACAECAQE/AVm3/8QAHhAAAgEEAwEAAAAAAAAAAAAAAREAAhIhMRAiUZH/2gAIAQEABj8CtpKE3w2saUwSV4J2fyXGouYm5//EABoQAQADAQEBAAAAAAAAAAAAAAEAESExUYH/2gAIAQEAAT8hGUTuxPYX2bBNVYjQYjwlL2lCcRCgUqy79n//2gAMAwEAAgADAAAAELgYvv/EABYRAQEBAAAAAAAAAAAAAAAAAAEAEf/aAAgBAwEBPxARiyUOWsLf/8QAFxEBAQEBAAAAAAAAAAAAAAAAAQARIf/aAAgBAgEBPxBB07AssL//xAAfEAEAAwEAAQUBAAAAAAAAAAABABEhQWExUYGxwdH/2gAIAQEAAT8QpmsQUx75mCCynZfDF+qGlae92Uz1A6jQA3XY1KOdafBNeXlr+ZHmV8B9QHat1X4z/9k=","width":325,"height":325,"src":"/static/e77add1988ced0aade2714bc66763b14/a2998/auto1-hackathon-builders-in-action.jpg","srcSet":"/static/e77add1988ced0aade2714bc66763b14/a2998/auto1-hackathon-builders-in-action.jpg 1x,\n/static/e77add1988ced0aade2714bc66763b14/99cce/auto1-hackathon-builders-in-action.jpg 1.5x,\n/static/e77add1988ced0aade2714bc66763b14/6e995/auto1-hackathon-builders-in-action.jpg 2x,\n/static/e77add1988ced0aade2714bc66763b14/968b1/auto1-hackathon-builders-in-action.jpg 3x"}}},"authorName":"Davide Debernardi","authorDescription":"Davide is Team Lead at AUTO1 Group, heading our Milan tech office.","authorAvatar":null,"headerImage":{"relativePath":"pages/one-more-hackathon-at-auto1/auto1-hackathon-the-engineers.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAALABQDASIAAhEBAxEB/8QAFwAAAwEAAAAAAAAAAAAAAAAAAAMEBf/EABYBAQEBAAAAAAAAAAAAAAAAAAIAAf/aAAwDAQACEAMQAAABqTnGKocG/8QAGxABAAICAwAAAAAAAAAAAAAAAQACBBMDETT/2gAIAQEAAQUCckDf3DlraHnyQJqo1//EABYRAQEBAAAAAAAAAAAAAAAAAAARIf/aAAgBAwEBPwGsf//EABURAQEAAAAAAAAAAAAAAAAAAAAS/9oACAECAQE/AVP/xAAdEAACAgEFAAAAAAAAAAAAAAAAAQIRIiEyQVGB/9oACAEBAAY/AtF4ZJG1Fka6I48H/8QAGxABAAIDAQEAAAAAAAAAAAAAAQARITFhUYH/2gAIAQEAAT8hLuR9ZInRHINaX2AKN3uA1VzB2haz/9oADAMBAAIAAwAAABCwP//EABgRAQEAAwAAAAAAAAAAAAAAAAEAQVGh/9oACAEDAQE/EAagzO3/xAAWEQEBAQAAAAAAAAAAAAAAAAAAARH/2gAIAQIBAT8Qrb//xAAaEAEAAwEBAQAAAAAAAAAAAAABABEhQTFh/9oACAEBAAE/EA1tjTwcSXqB2/O7ty0RGq7+7KgBUU97FM31i6Y9MMqu9n//2Q==","width":1280,"height":720,"src":"/static/c0deec50dd3823b86375f427ba85f823/966a5/auto1-hackathon-the-engineers.jpg","srcSet":"/static/c0deec50dd3823b86375f427ba85f823/966a5/auto1-hackathon-the-engineers.jpg 1x,\n/static/c0deec50dd3823b86375f427ba85f823/aa36d/auto1-hackathon-the-engineers.jpg 1.5x,\n/static/c0deec50dd3823b86375f427ba85f823/26d0b/auto1-hackathon-the-engineers.jpg 2x"}}}},"html":"AUTO1
\n#nobull #passionate #engineering
\n\nDay 1
\nDay 2
\nWrapping up
\nDatabase Transaction Isolation
\nWhat is transaction isolation level?
\n\n
\n\n
\n\n \n\n\nISOLATION LEVEL \nDIRTY READ \nNON-REPEATABLE READ \nPHANTOM READ \n\n \nREAD UNCOMMITTED \npossible \npossible \npossible \n\n \nREAD COMMITTED \n-- \npossible \npossible \n\n \nREPEATABLE READ \n-- \n-- \npossible \n\n \n\nSERIALIZABLE \n-- \n-- \n-- \nNow what?
\n@Transactional annotation which supports isolation attribute. If you do not specify the isolation level in the code explicitly, the default isolation level will be applied, which means that the default isolation level of the underlying datastore will be used.\n
\nSHOW VARIABLES LIKE 'tx_isolation'; -- on MySQLSHOW default_transaction_isolation'; -- on PostgreSQLReferences
\n","fields":{"slug":"/transactionisolation/","tags":["coding","sql","transactions"]}}},{"node":{"id":"c61acf9d-7746-503a-ba8f-cf96f4042096","frontmatter":{"category":"Coding","title":"Secure coding practices for NodeJS Web Applications","date":"2018-06-20","summary":"This post highlights various coding practices for securing a NodeJS web application against the most critical web attacks.","thumbnail":{"relativePath":"pages/securing-nodejs-applications/thumbnail.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFwABAAMAAAAAAAAAAAAAAAAAAAECBf/EABUBAQEAAAAAAAAAAAAAAAAAAAEC/9oADAMBAAIQAxAAAAHMkZoCv//EABUQAQEAAAAAAAAAAAAAAAAAAAAS/9oACAEBAAEFApSl/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BJ//EABURAQEAAAAAAAAAAAAAAAAAAAEQ/9oACAECAQE/AWf/xAAXEAADAQAAAAAAAAAAAAAAAAAAECEx/9oACAEBAAY/AoYv/8QAGhABAAIDAQAAAAAAAAAAAAAAAQARIUFxUf/aAAgBAQABPyHOqLFGlciV85P/2gAMAwEAAgADAAAAEHvP/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAh/9oACAEDAQE/EAJs/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFjP/8QAHBABAAEEAwAAAAAAAAAAAAAAASEAETFxUWHw/9oACAEBAAE/EMpW8q+4qwMgymddUsU0EV//2Q==","width":867,"height":325,"src":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/a2998/thumbnail.jpeg","srcSet":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/a2998/thumbnail.jpeg 1x"}}},"authorName":"Rajababu Pradhan","authorDescription":"Raja is Senior Frontend Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/securing-nodejs-applications/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/securing-nodejs-applications/header-image.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFwABAAMAAAAAAAAAAAAAAAAAAAECBf/EABUBAQEAAAAAAAAAAAAAAAAAAAEC/9oADAMBAAIQAxAAAAHMkZoCv//EABUQAQEAAAAAAAAAAAAAAAAAAAAS/9oACAEBAAEFApSl/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BJ//EABURAQEAAAAAAAAAAAAAAAAAAAEQ/9oACAECAQE/AWf/xAAXEAADAQAAAAAAAAAAAAAAAAAAECEx/9oACAEBAAY/AoYv/8QAGhABAAIDAQAAAAAAAAAAAAAAAQARIUFxUf/aAAgBAQABPyHOqLFGlciV85P/2gAMAwEAAgADAAAAEHvP/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAh/9oACAEDAQE/EAJs/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFjP/8QAHBABAAEEAwAAAAAAAAAAAAAAASEAETFxUWHw/9oACAEBAAE/EMpW8q+4qwMgymddUsU0EV//2Q==","width":1200,"height":450,"src":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/935ac/header-image.jpeg","srcSet":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/935ac/header-image.jpeg 1x"}}}},"html":"Secure coding practices for NodeJS Web Applications
\n1. Use SSL/TLS for communication
\n\n$ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365\nconst fs = require('fs');\nconst https = require('https');\nconst express = require('express');\n\nconst NODE_ENV = process.env.NODE_ENV || 'development';\nconst PORT = process.env.PORT || 3443;\n\nconst app = express();\n\nhttps.createServer({\n key: fs.readFileSync('/path/to/key.pem'),\n cert: fs.readFileSync('/path/to/cert.pem')\n}, app).listen(PORT);\n\n// Redirect http requests to use https in production\nif (NODE_ENV === 'production') {\n app.use((req, res, next) => {\n if (req.header('x-forwarded-proto') !== 'https') {\n res.redirect(`https://${req.header('host')}${req.url}`);\n } else {\n next();\n }\n });\n}2. Use security headers generously
\nStrict-Transport-Security: max-age=<expire-time>strict-transport-security:max-age=15552000;<frame>, <iframe> or <object>. Sites can use this to avoid clickjacking attacks, by ensuring that their content is not embedded into other sites.\nX-Frame-Options: DENY\nX-Frame-Options: SAMEORIGIN\nX-Frame-Options: ALLOW-FROM https://siteutrust.com/X-XSS-Protection: 1X-Content-Type-Options: nosniffContent-Security-Policy: policy\nconst express = require('express');\nconst helmet = require('helmet');\n\nconst app = express();\n\n<b>app.use(helmet())</b>\napp.use(helmet({\n frameguard: {\n action: 'deny'\n }\n}));\nadd_header X-Frame-Options DENY;\nadd_header X-Content-Type-Options nosniff;\nadd_header X-XSS-Protection 1;\nadd_header Content-Security-Policy \"default-src 'self'\";3. Preventing CSRF attacks
\n\nconst cookieParser = require('cookie-parser');\nconst csrf = require('csurf');\nconst bodyParser = require('body-parser');\nconst express = require('express');\n\nconst csrfProtection = csrf({ cookie: true });\nconst parseForm = bodyParser.urlencoded({ extended: false });\n\n// create express app\nconst app = express();\n\n// we need this because \"cookie\" is true in csrfProtection\napp.use(cookieParser());\n\napp.get('/form', csrfProtection, (req, res) => {\n // pass the csrfToken to the view\n res.render('send', { csrfToken: req.csrfToken() });\n});\n<form action=\"/process\" method=\"POST\">\n <input type=\"hidden\" name=\"csrf_token\" value=\"{{csrfToken}}\">\n\n Enter amount: <input type=\"number\" name=\"amount\">\n <button type=\"submit\">Submit</button>\n\n</form>4. Preventing XSS attacks
\n\nimport validator from 'validator';\n\nif(validator.isEmail('foo@bar.com')) {\n // Process email or store in DB\n}\nconst express = require('express');\nconst app = express();\nconst xssFilters = require('xss-filters');\n\napp.get('/', (req, res) => {\n let firstname = req.query.firstname; //an untrusted input\n res.send('<h1> Hello, ' + xssFilters.inHTMLData(firstname) + '!</h1>');\n});\n\napp.listen(3000);5. Preventing SQL Injection(SQLi) attacks
\n\n// SQL query vulnerable to SQLi\nsql = \"SELECT * FROM users WHERE username='\" + username + \"' AND password='\" + password + \"'\";\n\n// Execute the SQL statement\ndatabase.execute(sql)' OR '1'='1' --\nsql = \"SELECT * FROM users WHERE username='\" + ' OR 1=1 -- + \"' AND password='\" + password + \"'\";6. Secure cookies using cookie flags
\n\napp.use(session({\n secret: ‘My super secret’,\n <b>cookie: { httpOnly: true, secure: true }</b>\n}));7. Preventing brute force and DoS attack
\n\nconst RateLimit = require('express-rate-limit');\n\nconst limiter = new RateLimit({\n windowMs: 15*60*1000, // 15 minutes\n max: 100, // limit each IP to 100 requests per windowMs\n delayMs: 0 // disable delaying — full speed until the max limit is reached\n});\n\n// apply to all requests\napp.use(limiter);8. Error Handling
\n9. Tools of the trade
\n\n$ nsp check\n$ npm install -g snyk\n$ cd your-app\n$ snyk testFurther Readings
\n\"Hello, World!\"
\nprint \"Hello, World!\"
![](\"https://lh4.googleusercontent.com/8EZA89AIL5p8v723-rdyslTwOTbCIAqreXMF9_sd-fP6L7EEfbSN9h1yCvd6n7Ljc62P5GXAdj0KvtGqLhU2BKUdVd9dQEh8QP3rCRMhCZ2Q2yo_Rq9vrAj8grXr06wyWqhMYIAhtlgoG3uzqZGzCw8hcwhZHWXtvUgA7wamde5V5ZNnhkmwXMis0Q\")
![](\"https://lh3.googleusercontent.com/lxogAJ5Uq3wfk3-co_sG3dt_nnSph-bOuOOHItnd_-IKRcvOAzJ50PJgJnyRwKeHdwHk5pFWroeEbRuOcOcm-DiFCTKOKlCqsduwyxQK1nUknzKJpXPPv8z405le3v3C2oMqr2qYp1oq5Mj_XPP4cAhQDjYMUUPcSq-WP-W304PIw2ak2b5qnCNi0Q\")
![](\"https://lh6.googleusercontent.com/2pAyEXvc4hlAGNcMYNalJJDwM91YihSX4NdVRO-Hwzp52Zxldz_P0PH6OSlPSjp4DgGv5TSRTbjzxTQs3I0Li5h1_57g4tR9TB__80NL-petgqahPdZeOcgRYJt9G6h12xrLQfjmm9ijjJbTzXLqJPps3Ana2TZ5yTNJ_VXap4Hd6C7OZahMiHQY1g\")
![](\"https://lh3.googleusercontent.com/2xOP2Pq7LztBRRr-B2VJ5_roCubvYa4xJ_SlVeTz8LWzRTYqXflYowAAaNDsDh7y77M2vDDB8l1r_BpOvQsKQfHGssADO7FnbzGEmjtXKfFKfV5mES8whAQdYfUTJHwEdv9OGJ2A2GvkxS5XglrDG29p7zNZrtFNXvAMjThrQloKfFmSuqs7Nmd-xw\")
![](\"https://lh5.googleusercontent.com/v1PbZaRZQJjKJbBQiQZOXRUGjLK_c000j3edWpNb7jhim2dDHl21ircQVV4-aabJnHA57-A68J7WXfTkMFFxeVro0kGAlSMh2PKZ72yW5IezNvN-FiM4Xm-siC7PhAS8ZarnG4imewzKJhjKxAR7Mt0A3R5BOkpSCDYW14ldqG2CLpW5XC-jbwJtvg\")
![](\"https://lh3.googleusercontent.com/FddBhrlKYHmv4Jw9mucH_Mzo0OUs7kbDEN2xOLqfPm7NCyNcM0fxD0MFSSQHODLDWYL8x-CrwCAj-nTJdeqbZGNk-5MrRwxrg7FI4ILuTQTQe8uEPG7OqHLrRZHnuhnjC9Os3x7KcBEDCaQWGJKk_amQo8GnfBh4PK87vfL6cgGD8buX61uiYEIclQ\")
![](\"https://lh6.googleusercontent.com/mEaBTCaWa1lWEc0cjQraEWVOPMb3V8StdxwrMvaALgxSxMBfjNjk4KBCRif-F_2oUVzgCBheGOpyLiAK5tnxPcH1m2Nae-8EMyi3CgOhoMUSSVAUJY8jFyckMLAq1PMhPI6UPE_uHSEeCy4yNrpX-mPx4he1VywC5lCtkwJBaC77Px36aqiMivkwig\")
![](\"https://lh3.googleusercontent.com/9ofjHjm_CHNEKWH-7C6kdE5aq9ELbeH4ekpOoOZcDMJtwYjP3xWOTiceE3kW1i0pl4H3uTCbh0c5eY1MMqpWJRXjB0J4L66x0vf6YHhs15fY-s_oA-8qobF_eynejn8xlsns09OlmgIGAy7bZNDDebFodPEHmGrPgp8HnAhoCCUIzEi7fV1OWheP\")
![](\"https://lh4.googleusercontent.com/x_T21RH2LmaVQypUzhNgve4ZEPeKBTvWs0IXdJdPF_RVz3EXAb3SK09ZR1JRxmt6OUSTQdRpowIdPGUI0A17-ATiLE-nSY3u8hA0OjZZMO9HF-RCBw9q0I5GG_xIEwMyMy2g5OpxWc7QTmN2KnZy29jJ8pTdt0Av3MNQ4FuKS0e7FT3aRuYZv1diWA\")
![](\"https://lh3.googleusercontent.com/HsXSzSFwhynsa77tvxtyWg2K54pJpTXvXMghLmselpYCFLk7vktBkwWlQ5q7T_WpTiEYSTi85ULtxxz5fNhfleOVEhXf5NRm_51YPAh9x_t1aI53KyHs8QA1bRqLuhjMtZeSXwQIRrOq8rI2FxIxFs2bktIuLclo1ZgQN4sGEcdEKrrW-ZzE10tRHQ\")
![](\"https://lh5.googleusercontent.com/qyj0MbKLCEPO9M3Q8kalZoXkgfgcxddnmxk6lYUu0DrbBHhT747lCyy1SFhZDHKZXOGwUqcnsDFDPM2_rWAhnWC5l5m58RVcpsWWrBmwHIZgd3tKxSmhC6aqZHsB016Z7MXAhT0EX7mnT_N3_w\")
![](\"https://lh6.googleusercontent.com/F4M8xRSQWe86JzTN8G5uPIc62Y_uNhPB-_YDLlhHtTpy0PseDIX7fb8_jh37YbpVQEIJiy4gE76vP0eqkU5jjHKOeac5h3tprc52Iz2oQvv0Ros5AHC00DqsW_YZ6H_hXWdJ0yPe6evaU3Ir7Q\")
![](\"https://lh3.googleusercontent.com/yvtgM7DEm071JAOlv7stZjfjb3j31gbUhlMFH7mdXrls0TMaBG7FlV8kU6hgo99AGZ6LxotbaGzkZjbD6EiYfjqhj2j4uEQmR3n0qE6rwFl0XbPOfVgpcMabcdn4Zvh-pQH8jOt9\")
![](\"https://lh6.googleusercontent.com/Cr_GOw3whsmE5QUmFqElFtGzwUYhAudHXrkpI5BBwgWSgsD6DI1C1IDgO9ImhQTXjaBbd13IUntuxNY7wRBvBw1wGpYl7rl5bfk39ICXz8e_vMFnghBi530Oou9unORSShG7dJ9f\")
![](\"https://lh3.googleusercontent.com/mQt3PeJM0OMd1UkEF_iYvz6LSyr7BWbafyqyjxHPaVyYfebMBD00VXmIM8R_acox5DKYEN4OAc-LRRu8feSl6ERgKxwYJGHIPAg1U7zUzYvmj6s-SSnA4DTT2ENnXvPiV_xA40GQ\")
![](\"https://lh6.googleusercontent.com/KyLHXVRSIowTRybMH7hAOgiCvdzGqNBqg57UYB9lQ4K-zskAPsVG_e7hoKVAaM3qJ4F3Xh8Yshx65yCF0e_n3LyM2AZ5MyPBnLuZFdyDfg-5JDRF9-24zXMSQyRrBel70VoGJJW7\")
![](\"https://lh5.googleusercontent.com/ET-G6wXxbo6O1YniBzp4lRQU6D7DAo9WPn0ugiHifyGVV3-E6KV3Gf-ETvwYYnE3MlTeQlIkPp3EtxxL8BSB-vgVjn0-34UBUE2C2_y_lCKLtgWQHdhEWAmQDOHqQX-BDZPYwpE8\")