![](\"https://lh6.googleusercontent.com/F4M8xRSQWe86JzTN8G5uPIc62Y_uNhPB-_YDLlhHtTpy0PseDIX7fb8_jh37YbpVQEIJiy4gE76vP0eqkU5jjHKOeac5h3tprc52Iz2oQvv0Ros5AHC00DqsW_YZ6H_hXWdJ0yPe6evaU3Ir7Q\")
If you are already familiar with MDC (Mapped Diagnostic Context) you can go to the next section of this article. If not, please take a look at this first to get a better understanding of what MDC is.
\nIn one of our services we had an endpoint responsible for invoking process related to car auction with a lot of logs for debugging purposes. We already had a request-id to trace it in Kibana, but we wanted to filter logs in a more convenient way, by value that we know even before invoking this endpoint which was the stock number of the car. So we decided to use MDC as it allows us to add diagnostic key-value pairs to every log in given Thread.
\nAlthough MDC is a great tool, it forces programmers to write clunky code due to the fact that every key-value pair that was set should be cleared, otherwise in some edge cases it could be used by another thread.
\nClearing can be done by invoking MDC.clear() or by MDC.remove(key). First one clears all MDC keys so we could accidentally remove keys that were set higher in stack. Second one is better then but it has to be invoked for every key. And also in both cases it should be done in finally block to be sure that no matter what happened, context was cleared.
\ntry {\n MDC.put(\"key1\", \"value1\");\n MDC.put(\"key1\", \"value2\");\n //some process\n log.info(\"very important log\");\n}finally {\n MDC.remove(\"key1\");\n MDC.remove(\"key2\");\n}MDC also has method putCloseable(key, value) which returns AutoClosable object but it allows only one key-value pair and has to be assigned to a variable, which may look better for a single key but it is not perfect. IDE will complain about never used variable
try (MDC.MDCCloseable closeable = MDC.putCloseable(\"key\", \"value\")) {\n //some process\n log.info(\"very important log\");\n}To stand against MDC drawbacks, we came up with this functional style solution:
\nLogContext.withLogKey(\"key1\", \"value1\")\n .andKey(\"key2\", \"value2\")\n .andKey(\"key3\", \"value3\")\n .execute(() -> {\n // some process\n log.info(\"very important log\");\n });At first glance we can see that there is no Try Catch block anymore and we are only setting key-value pairs and not clearing them, however everything works exactly the same. So how is it done?
\nSolution is pretty simple. It is based on a container class for storing MDC context keys mixed with Step Builder Pattern. Let's take a look at implementation.
\n@NoArgsConstructor(access = AccessLevel.PRIVATE)\npublic final class LogContextSteps {\n\n public interface ContextStep {\n ExecuteStep andKey(final String key, final Object value);\n }\n\n public interface ExecuteStep extends ContextStep {\n\n <T> T execute(final Supplier<T> action);\n\n default void execute(final Runnable action) {\n if (action == null) {\n throw new IllegalArgumentException(\"Action for execution cannot be null.\");\n }\n\n execute(() -> {\n action.run();\n return null;\n });\n }\n }\n}LogContextSteps is a wrapping class for two step interfaces:
\nContextStep which contains method for adding new log key-value pair
\nExecuteStep which also extends ContextStep and serves execute method responsible for invoking action that we want to wrap with MDC.
\npublic class LogContext implements ContextStep, ExecuteStep {\n private static final MDCLogContextSetter MDC_LOG_CONTEXT_SETTER = new MDCLogContextSetter();\n\n private final LogContextSetter contextSetter;\n private final Set<String> contextKeys = new HashSet<>();\n\n private LogContext(final LogContextSetter contextSetter) {\n this.contextSetter = contextSetter;\n }\n\n public static ExecuteStep withLogKey(String key, Object value) {\n return withContextSetter(MDC_LOG_CONTEXT_SETTER).andKey(key,value);\n }\n\n static ExecuteStep withContextSetter(LogContextSetter contextSetter){\n return new LogContext(contextSetter);\n }\n\n @Override\n public ExecuteStep andKey(String key, Object value) {\n contextSetter.put(key, value);\n contextKeys.add(key);\n return this;\n }\n\n @Override\n public <T> T execute(final Supplier<T> action) {\n if (action == null) {\n throw new IllegalArgumentException(\"Action for execution cannot be null.\");\n }\n\n try {\n return action.get();\n } finally {\n contextKeys.forEach(contextSetter::remove);\n }\n }\n\n interface LogContextSetter {\n\n void remove(final String key);\n\n void put(final String key, final Object value);\n }\n\n private static class MDCLogContextSetter implements LogContextSetter {\n\n @Override\n public void remove(String key) {\n MDC.remove(key);\n }\n\n @Override\n public void put(String key, Object value) {\n MDC.put(key, Objects.toString(value));\n }\n }\n}LogContext is our main class here. It:
implements interfaces from LogContextSteps
stores MDC keys in Set
executes action that we want to wrap with MDC context
\nFor testing purposes there are some package protected classes and methods like LogContextSetter interface and its implementation MDCLogContextSetter, as it would be hard to test static methods of MDC class.
So, how does this flow work?
\nWe begin with the static factory method LogContext.withLogKey(\"key1\", \"value1\") which sets up our MDCLogContextSetter and invokes the andKey(\"key2\", \"value2\") method for the first time. Then we can chain next invocations of andKey where every invocation causes two things:
MDC key is stored in Set<String> contextKeys as only keys are needed to clear context
LogContextSetter::put method is invoked where implementation is: MDC.put(key, Objects.toString(value)) and at that moment key-value is stored in MDC
Finally we can invoke execute(() -> action()) which is classic try-finally block where try invokes our action and finally clears MDC iterating every through contextKeys and invoking LogContextSetter::remove where implementation id MDC.remove(key)
Assuming that we have an ELK stack for monitoring and proper log configuration we can go to Kibana to find interesting logs via our MDC key-value pairs. Depending on the log configuration, key can be directly accessible or under the flat field (which was in our case)
\nFor key-value pair: stock-number - SN12356 Kibana filter may look like this:
\nIntro
\nIn this article, we will take a look at spring-batch and how it could be used. We will walk through various configurations and we will create an application which reads from a CSV file & writes into a database with outstanding performance.
\nI used the following: Java 11, Spring 5+, Spring Boot 2 and Maven-based project.
\nCreate a Project
\nFirst, we need to create a Spring Boot 2 project. We recommend that you do so by visiting spring initializer, which is a useful tool to generate spring projects with required dependencies and configurations.
\nDependencies
\nWe need the dependencies below to run and test the project:
\n<dependency> \n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-batch</artifactId>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-data-jpa</artifactId>\n</dependency> \n\n<dependency>\n <groupId>org.projectlombok</groupId>\n <artifactId>lombok</artifactId>\n <version>1.18.10</version>\n <scope>provided</scope>\n</dependency>\n\n<dependency>\n <groupId>com.h2database</groupId>\n <artifactId>h2</artifactId>\n</dependency>\n \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n <exclusion>\n <groupId>org.junit.vintage</groupId>\n <artifactId>junit-vintage-engine</artifactId>\n </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.batch</groupId>\n <artifactId>spring-batch-test</artifactId>\n <scope>test</scope>\n</dependency>\n\n<dependency>\n <groupId>org.hamcrest</groupId>\n <artifactId>hamcrest-all</artifactId>\n <version>1.3</version>\n <scope>test</scope>\n</dependency>spring-boot-starter-batch dependency includes all the configurations to run the spring batch application. Lombok is just a helper dependency to write the code faster and cleaner. H2 is used as an in-memory database. spring-boot-starter-test and spring-batch-test are included for test purposes.
\nBook Class
\nLet’s create a model class book, which will represent a book. This will just serve as a model class and will help us during implementation of spring batch.
\n@Data\n\npublic class Book {\nprivate String title;\nprivate String description;\nprivate String author;\n}Configuration
\nLet’s create a class called SpringBatchConfiguration. We will add all required configurations here. First, let’s annotate this class with @Configuration to be able to inject beans, and with @EnableBatchProcessing to enable spring batch processing. Additionally, we can add @RequiredArgsConstructor from Lombok which would help us to generate constructor with parameters. These parameters are the ones which are marked as final class properties. These properties will be injected as spring beans. Our class will be like this:
\n@Configuration\n@EnableBatchProcessing\n@RequiredArgsConstructor\npublic class SpringBatchConfigurationNow, in SpringBatchConfiguration class let’s add properties which need to be injected into the constructor:
\nprivate final JobBuilderFactory jobBuilderFactory;\nprivate final StepBuilderFactory stepBuilderFactory;jobBuilderFactory and stepBuilderFactory are declared in spring batch jar as spring beans so that we can inject them in any class we want.
\nFile Reader
\nNow, we need to declare and initialize spring beans to configure the batch process. The first bean which we will need will be responsible for reading from a file line by line. Spring Batch provides a default class for it. The class name is FlatFileItemReader. Similarly, spring has different default reader classes for reading from a relational database, mongodb and etc. However, if you need, you can create your own reader and implement it in a way you want.
\nLet’s now see what FlatFileItemReader injection will look like.
\n@Bean\n@StepScope\npublic FlatFileItemReader<Book> bookReader(@Value(\"#{jobParameters['filePath']}\") final String filePath\n) {\nreturn new FlatFileItemReaderBuilder<Book>()\n .name(\"personItemReader\")\n .resource(new ClassPathResource(filePath))\n .delimited()\n .names(new String[]{\"title\", \"description\", \"author\"})\n .fieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {{\n setTargetType(Book.class);\n }})\n .build();\n}We are configuring that the bean reader should read data from the given file path, which should be a CSV file having rows with title, description and author respectively.
\nThe StepScope means to initialize this bean after each step, so file path could be dynamically set when the spring batch job is launched. We will come to that later, how to pass the file path later in this article.
\nItem Writer
\nNow, let’s create a writer class, which will take the data and write into the relational database.
\n@Bean\npublic JdbcBatchItemWriter<Book> writer(final DataSource dataSource) {\n return new JdbcBatchItemWriterBuilder<Book>()\n .itemPreparedStatementSetter((book, preparedStatement) -> {\n preparedStatement.setString(1, book.getTitle());\n preparedStatement.setString(2, book.getDescription());\n preparedStatement.setString(3, book.getAuthor());\n })\n .sql(\"INSERT INTO books (title, description, author) VALUES (title, description, author_surname)\")\n .dataSource(dataSource)\n .build();\n} In the writer bean, we are setting item processors and adding values inside preparedStatement accordingly, which book property should be inserted for each db table column.
\nStep Configuration
\nNow, let’s configure a step which will be executed in the batch process. In our case, the configuration will look like this:
\n@Bean\npublic Step step1(final ItemWriter<Book> writer, final ItemReader<Book> reader) {\n return stepBuilderFactory.get(\"step1\")\n .<Book, Book> chunk(100)\n .reader(reader)\n .processor((ItemProcessor<Book, Book>) book -> book)\n .writer(writer)\n .build();\n}In our scenario, we have only one step, but it's also possible to configure multiple steps. Here, we are creating a spring batch step with the name step1 and setting reader and writer accordingly. These are the readers and writers which we created as spring beans earlier.
\nWe are setting chunk as 100, which means the items chunk proceeded is 100. We can make this configurable too. The processor is for converting the current object to the one which the writer should proceed with. In our case, it is the same object.
\nJob Configuration
\nLast but not least, let’s configure the job which should be executed.
\n@Bean\npublic Job importUserJob(final Step step1) {\n return jobBuilderFactory.get(\"bookReaderJob\")\n .incrementer(new RunIdIncrementer())\n .flow(step1)\n .end()\n .build();\n}Here, we create a job with the name bookReaderJob. We add an incrementer RunIdIncrementer, which is an id generator for the tables which are specifically designed to store the details about job executions in the database. After each time the spring batch job is executed, it will save details about the execution. To check the schema structure for storing this data, take a look at this sql. It is for MySql, but other SQL scripts are available too.
\nAdditionally, we add flow by which the job should be executed. Currently, we have only one step in our flow, so we add it.
\nPlease also add this config: spring.batch.job.enabled=false in your properties config file so that spring batch job won’t be executed automatically with no parameters when the application is started.
Execution
\nTo execute the job, we need to declare JobLauncher and launch the job. To do so, we need to create the below class:
\n@Component\n@RequiredArgsConstructor\npublic class SpringBatchExecutor { \n private final JobLauncher jobLauncher;\n private final Job job; \n\n @SneakyThrows\n public void execute(final String filePath) {\n JobParameters parameters = new JobParametersBuilder()\n .addString(\"filePath\", filePath)\n .toJobParameters(); \n jobLauncher.run(job, parameters);\n } \n}Now, we can call the execute method from whenever we want, with the file path from which the data should be read.
\nAdditional Classes
\nThere are a few additional classes which you will need to execute your code.
\nCreate a class BookEntity so that spring data JPA will automatically create the book table for you. Then you can create repository.
\n@Data\n@Entity\n@Table(name = \"books\")\npublic class BookEntity {\n @Id\n @GeneratedValue(strategy = GenerationType.IDENTITY)\n private Long id;\n private String title;\n private String description;\n private String authorFullName;\n}Then create the interface BookRepository and extend it from JpaRepository. At the moment, we will need this only for testing.
\n@Repository\npublic interface BookRepository extends JpaRepository<BookEntity, Long> {\n} Testing
\nNo one likes a code which is not tested. So, let’s write a few test cases for our class.
\nHere is a code sample for test cases:
\n@Slf4j\n\n@SpringBootTest\nclass SpringBatchSampleApplicationIntegrationTest {\n\n @Autowired\n private SpringBatchExecutor springBatchExecutor;\n \n @Autowired\n private BookRepository bookRepository;\n\n @Test\n public void testExecution() {\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"sample-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(7L));\n } \n\n @Test\n public void testLargeData() {\n long startTime = System.currentTimeMillis();\n long initialCount = bookRepository.count();\n assertThat(initialCount, equalTo(0L));\n springBatchExecutor.execute(\"large-data.csv\");\n long count = bookRepository.count();\n assertThat(count, equalTo(60000L));\n long endTime = System.currentTimeMillis();\n log.info(\"executed in miles: {}\", endTime - startTime);\n }\n}The second test executes in ≈ 3500 ms in a machine with a 2.2 Ghz Intel Core i7 and 16GB ram. It proceeds 60K lines of CSV file, and saves it into a relational database.
\nYou can check out the working sample code in github,
","fields":{"slug":"/spring-batch-integration/","tags":["auto1","engineering","spring","code","java"]}}},{"node":{"id":"e2563427-db60-5c74-aaff-0c8c2c483f69","frontmatter":{"category":"Engineering","title":"DevoxxUA was awesome!","date":"2019-12-05","summary":"A few words about one of the biggest Java conferences and AUTO1's presence there.","thumbnail":{"relativePath":"pages/auto1-devoxx/ivan_thumb.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAABQABBP/EABUBAQEAAAAAAAAAAAAAAAAAAAEA/9oADAMBAAIQAxAAAAFobuDFqyL/xAAaEAABBQEAAAAAAAAAAAAAAAACAAEDEzES/9oACAEBAAEFApMKYRldXWI3bvF//8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BJ//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8BP//EAB0QAAEEAgMAAAAAAAAAAAAAAAABERIhAxBCUYH/2gAIAQEABj8C9H496SmExxqQx//EABsQAAMAAgMAAAAAAAAAAAAAAAABETFRIUGh/9oACAEBAAE/IWihpe8JGzIwO65PagSFoof/2gAMAwEAAgADAAAAEEgv/8QAFREBAQAAAAAAAAAAAAAAAAAAEDH/2gAIAQMBAT8Qg//EABYRAQEBAAAAAAAAAAAAAAAAAAEQIf/aAAgBAgEBPxBNn//EAB4QAQACAgIDAQAAAAAAAAAAAAEAESFBMWFRcbHh/9oACAEBAAE/EPcQfYt7YYNab5CuSAbFIzJ3xnp6mFDht8n9YZ1dVqq6n//Z","width":520,"height":325,"src":"/static/150f8a5d8b98da0a021f785cc45b217a/a2998/ivan_thumb.jpg","srcSet":"/static/150f8a5d8b98da0a021f785cc45b217a/a2998/ivan_thumb.jpg 1x,\n/static/150f8a5d8b98da0a021f785cc45b217a/99cce/ivan_thumb.jpg 1.5x,\n/static/150f8a5d8b98da0a021f785cc45b217a/6e995/ivan_thumb.jpg 2x,\n/static/150f8a5d8b98da0a021f785cc45b217a/968b1/ivan_thumb.jpg 3x"}}},"authorName":"Ivan Kozlov","authorDescription":"Ivan is the senior team lead of the AUTO1 Group Kiev engineering office.","authorAvatar":{"relativePath":"pages/auto1-devoxx/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/auto1-devoxx/header.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAJABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAIDBf/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAe21ELH/xAAZEAACAwEAAAAAAAAAAAAAAAACEQABEiD/2gAIAQEAAQUC0TZSuP/EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQMBAT8BP//EABQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8BP//EABcQAAMBAAAAAAAAAAAAAAAAAAAgMTL/2gAIAQEABj8CyRf/xAAaEAEAAgMBAAAAAAAAAAAAAAABADERIFEh/9oACAEBAAE/IaFO5g3PlzMEdP/aAAwDAQACAAMAAAAQAA//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAEDAQE/ED//xAAUEQEAAAAAAAAAAAAAAAAAAAAQ/9oACAECAQE/ED//xAAbEAADAAIDAAAAAAAAAAAAAAABESEAMSBBYf/aAAgBAQABPxClqP0uMAdzbvAsCMJhXD//2Q==","width":640,"height":284,"src":"/static/cae90168dbe3b194c3406bc103d1a5f3/c8462/header.jpg","srcSet":"/static/cae90168dbe3b194c3406bc103d1a5f3/c8462/header.jpg 1x"}}}},"html":"Devoxx, formerly known as JavaPolis, is an international community conference which first took place in 2001 in Belgium. Since then it has developed into one of the largest Java events in Europe, with chapters in Belgium, France, UK, Poland and Ukraine, as well as an established conference in Morocco.
\nAmong the speakers at DevoxxUA (Devoxx Ukraine) 2019 I’d like to mention Josh Long, the Spring Developer Advocate at Pivotal and Cay Horstmann (yes, that’s him, the author of “Core Java”, “Big Java” and other great books). I got a chance to participate as a speaker on the second day of the conference, delivering a presentation on one of our latest solutions for implementing GraphQL with authorization in a microservice architecture. Was very excited to see that the talk got the highest amount of participants for that time slot and the room was packed, including people standing on their feet for the entire duration of my presentation!
\nAll in all, it was a great experience of sharing what & how we’ve built while getting valuable feedback and interesting questions. Have to say it felt incredible to stand on stage next to all the stars of the engineering world, during the closing ceremony of the conference!
\nHere's my entire talk:\n\n
","fields":{"slug":"/auto1-devoxx/","tags":["auto1","engineering","java","conference","talk","devoxx"]}}},{"node":{"id":"7efdac28-b97f-53e0-adf8-48d022f3eecc","frontmatter":{"category":"Coding","title":"Integration Test Speedup with Spring","date":"2019-03-11","summary":"Here is what we found on speeding up your integration tests in spring based applications.","thumbnail":null,"authorName":"Boris Faniuk","authorDescription":"Boris is a former member of Engineering of AUTO1 Group.","authorAvatar":{"relativePath":"pages/integration-test-speedup/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/integration-test-speedup/header.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAPABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAABQABAv/EABYBAQEBAAAAAAAAAAAAAAAAAAIAAf/aAAwDAQACEAMQAAABJ6Uw4FJSv//EABsQAAICAwEAAAAAAAAAAAAAAAECAAMEFCEx/9oACAEBAAEFAl9tuawzH6WUPNYT/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQIBAT8BJ//EABsQAAEEAwAAAAAAAAAAAAAAAAABEBIhIjFR/9oACAEBAAY/AjJo9SizZ//EABwQAQEAAgIDAAAAAAAAAAAAAAERAHEQMUFR0f/aAAgBAQABPyFMdusqfAgeuBQwbbGHwF1jT4z/2gAMAwEAAgADAAAAELsf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARBB/9oACAEDAQE/EA2f/8QAFxEAAwEAAAAAAAAAAAAAAAAAAAERIf/aAAgBAgEBPxDSpD//xAAbEAEBAQADAQEAAAAAAAAAAAABEQAhMXFBYf/aAAgBAQABPxAMiBZ241OQL4wZbyughIofpPEuZhdFS61EeZ//2Q==","width":1024,"height":768,"src":"/static/9f60dcc74069c89c91db86dc33946167/9f594/header.jpg","srcSet":"/static/9f60dcc74069c89c91db86dc33946167/9f594/header.jpg 1x"}}}},"html":"Over time we realized that our microservices built on top of Spring framework take tons of time on the integration test step. Every single PR runs all tests and this takes 10-20 minutes for some services. To decrease PR build time we decided to research why this happens.\nOne of the problems that this research highlighted is ramping up heavy spring context several times during execution of all module’s tests.
\nAs a proof of concept we have taken 3 modules and cleaned up their tests to achieve better integration test performance.
\nThe results are pretty much self-explanatory:
\nBelow I list all factors that produce many spring contexts and how we managed to overcome them.
\n@ActiveProfiles and @ContextConfiguration with different attributesOur spring-based microservices actively use spring profiles and configuration classes.\nMany components from common libraries are hidden under profiles and configuration classes to not be active for every single module / environment without need. So, integration test authors define profiles and configuration classes that should be activated for their tests using these annotations:
\n@ActiveProfiles({\"integration-test\", \"pg-test\", \"mongo-test-two-nodes\"})@ContextConfiguration(classes = {Application.class, AwsTestConfiguration.class})Here mongo-test-two-nodes activates special mode for mongo based services to operate with two separate mongo instances. As not all tests need this, the profile is turned off by default and only those tests that need this behaviour would explicitly turn it on. But this, unfortunately, leads to multiple spring contexts being created.
\nThe same happens with AwsTestConfiguration that deploys special test-component to mock AWS calls.
\nThe solution was to make all tests use the same set of active profiles and context configuration classes for every single integration test.\nThe drawback is that some components are deployed, even for the tests that do not need it. But deploying single component takes O(1) time comparing with the whole spring context.
\nAnother point to keep in mind is that order of profiles (configuration classes) matters.\n@ActiveProfiles({\"integration-test\", \"pg-test\"})\n@ActiveProfiles({\"pg-test\", \"integration-test\"})
Two settings mentioned above result in 2 different spring contexts.
\n@SpyBean and @MockBeanOne interesting feature in spring-boot-test is mocking / spying real spring beans.\nThis was mostly used when we don’t care what some method does in test but we need to check that it was called by parent components.\nUnfortunately this also makes test to create separate spring context.\nWhat we’ve done here is replacing @SpyBean annotation with configuration class that wraps real spring bean into Mockito.spy(..):
@Configuration\npublic class SomeComponentSpyFactory {\n\t@Autowired\n\tprivate SomeComponent someComponent;\n\n\t@Bean\n\t@Primary\n\tpublic SomeComponent someComponentSpy() {\n \t\treturn Mockito.spy(someComponent);\n\t}\n}Another option is to use the combination of @Autowired and @Spy annotations.
\t@InjectMocks\n\t@Autowired\n\tprivate ParentComponent parentComponent;\n\n\t@Spy\n\t@Autowired\n\tprivate ChildComponent childComponent;However, the drawback of the last solution is that, parentComponent needs to have a setter for childComponent which is not good in terms of encapsulation. But if you don’t use autowiring on constructor level it is fine. Anyway, one can choose whatever fits the situation better.
\n@DirtiesContextOne of the services used this annotation to clear the caches between different tests.\nClearing a cache is really a problem (see section below), but this solution comes to separate spring context per test class or even test method (depending on annotation parameters). Context per method was actually the case for a few test classes. Instead in this modules we added methods to clear the caches programmatically without deploying the whole new context.\nOne can use java reflection API instead of adding special production-level methods, again whatever fits you better.
\nAfter all this “multiple context” factors are gone one may face (and we did) a challenge to fix or re-organize some tests. Before above enhancements were done all tests were running in separate spring contexts, therefore used separate instances of every component and therefore those problems could be hidden.
\nAs an example some tests in mongo-based service were testing getAll()-like operations. Those tests were running functions that fetched all records from some collections. This was not a problem as embedded mongo DB was different instance. After all tests started to exploit the same spring context some of them started to fail intermittently. The failures were intermittent because test execution order is different. One service may break another, but not vice versa.\nSo, to overcome this, we added missing @Before operations that cleaned those collections.
The rule that I would suggest here is that every test should clean up whatever it needs to be clean and not rely on fairness of other tests. That was a problem in our case. Some tests had @After operations that cleaned up, but not everything produced by the test was cleaned up. This caused intermittent failures of other tests.
So, now after we know all this, we created two tools (and may create more in future) that help to identify those problems.
\nAfter every PR commit we run one additional step that analyzes test execution log.\nIf more than one spring context is created this leaves some traces and a notification is sent to the author of the commit via slack
\nAs an enhancement, the jenkins jobs could be modified to fail the build if the number of spring contexts is greater than a certain threshold. The threshold could start with a high number and can gradually be reduced as the team cleans up the existing tests by adopting the aforementioned practices/fixes
\nWe wrote an IntelliJ plugin to identify and solve the aforementioned problems. There will be a separate blog post about this plugin coming very soon.
","fields":{"slug":"/integration-test-speedup/","tags":["auto1","engineering","spring","java","testing"]}}}]}},"pageContext":{"slug":"/tags/java","tag":"java","categories":["Architecture","Coding","DevOps","Engineering","ProjectManagement","QA","Social","TechRadar"]}}