{"data":{"allMarkdownRemark":{"edges":[{"node":{"id":"f3a0fb4b-eb2f-5624-a408-d63e3cee5184","frontmatter":{"category":"Coding","title":"Spring batch integration","date":"2020-02-07","summary":"Usage cases for spring-batch.","thumbnail":null,"authorName":"Artur Yolchyan","authorDescription":"Artur is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/spring-batch-integration/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEfElEQVQ4yz2O2VMbBQDGlzJQSylXIEc3F0eODYGEJAQI5GBzbK49kt0cm703CSRQqfQCB9taap1qgY61VXqoPZxxbKe2dWo7tY6d0RdHX/0HfPLRF991qTPO/OZ7+33fB+RDFjruKMWhcgwqwnYqaiVhKwXbiolRgZiuVxJ1LrsgEg25sFilFqrFBblQFXNVgajyBMCkxiuIs4yMMoiDSTiwoC3kc/hdVpdVPzFqRua9MpupS7mqmK9LVF0u1CVFzssCIXMYwKZcTGpMyWLUOe2yjoxAQyNO1YCutbV9f1tbe1u7dcRY47GaMiXuVVSFvMTjModLLApwaRefHqeibqNm4EBnb/+A1mo2T9kHcZ9tGR5LQocPvdFmNusEOqv8lPicxBESi4kstifzabeMuj1Wnba7i/Q71vDgJ/Xc4zXhyWrxq1pyG/eWvEMd+9umfWMLAiFWshKD/g8goROV5LjdBJa9ls9r2edr4svTjRenG8836o9WK7e46EXM5zbpNGoVX0qKdEqk00qFgkBnAGrOko+Nz1jA94iZhyvFZ8fZ707JLzaWft48/vJ08/6b5Y+K4azHph5QlbCIXE7yJYRXco8U4Ffvn7ZoAiPgRSp8vU59s9Z89s7RJ2fXtxerbxGpo5nIOWxaCLr1BhOZDUqlOF+ICyVEQSwnAcuBFmtHi8ekRT0OMjBZQyI3lvgduVCLzvGx0HJi9nw+WPTbJ4dNdSLMUTBXiAnF+GsSgKN7n62zxWnQVAIuOjCxEJnaxuevleMfkMjJzPxGcuZTPs4FoA0U3uHIIhHkqKgCr1CIAx6wBeptTY3ZTiRnzmPhbTp9gyFuc+hNBnm6Kj48tXy1Ri7Brs8WxTMVvITN8fkIT8I8GRXIKFCAR11g1wSo3WSIb9dXbojEnUb1C5m5XkJ+ePvI4/WVHSm3BE98SKNCZpbBg6xyntzz92QxN+U39XYBQDMdfbTWvCpiF/Kpyyy1Wy/fapS+XC5tcclq0JGdMOViHhYP/Sdzyn5+HuBR3+xwXwcABG1DNxvkOgqfLxR3q4vvoujFQvzeSuGymKV8w5C+m4p7GTTIYEE2F+b2iABcZnLW2tsNAPrOg5f4zJVyYgfH7vHibY76epm8eyR/BpuU0inPNBFwW6RcSPGVfQYPcbl5gM14Ex69/gDQvW8f7rHuyujtGnm3yT84Id45Sp8lw0dwZHPrqX0s3N/Xnwl56UyAI0KvCQN02lOIOcfV7erWFvBgOwwZj+VSm0LleDlfR+aOsfSDV3+yS1udHV2aAZNaBQ4aByddTjQ6VcGCAJv1wX472K9S96gOHeyZmkHoWJiJ+Jsxz9ax1Ve//33p3k86tVmnM2s0JgWtdlCrHdKDw5AVAoio+7DW2NOj6+lRazQj71/99crayY8b+P3rd3/5459Hv/017o7otENGo82gt5jNo0rq9cMGg0UPWoBZj7O3Dxzo1/f1alyT6OL645Xm2cvXflw+93Rz93tx6YJaZRocHDMYIKMRAg9bQHDYZIIMBrvZ7PwXcPlkRtKKCQsAAAAASUVORK5CYII=","width":50,"height":50,"src":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png","srcSet":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png 1x,\n/static/dc9516b9327e627890a1f5c7f84c004a/eb85b/avatar.png 1.5x,\n/static/dc9516b9327e627890a1f5c7f84c004a/4f71c/avatar.png 2x,\n/static/dc9516b9327e627890a1f5c7f84c004a/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"

Intro

\n

In this article, we will take a look at spring-batch and how it could be used. We will walk through various configurations and we will create an application which reads from a CSV file & writes into a database with outstanding performance.

\n

I used the following: Java 11, Spring 5+, Spring Boot 2 and Maven-based project.

\n

Create a Project

\n

First, we need to create a Spring Boot 2 project. We recommend that you do so by visiting spring initializer, which is a useful tool to generate spring projects with required dependencies and configurations.

\n

Dependencies

\n

We need the dependencies below to run and test the project:

\n
<dependency> \n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter</artifactId>\n</dependency>\n   \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n    <exclusion>\n     <groupId>org.junit.vintage</groupId>\n      <artifactId>junit-vintage-engine</artifactId>\n    </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-batch</artifactId>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-data-jpa</artifactId>\n</dependency>    \n\n<dependency>\n <groupId>org.projectlombok</groupId>\n <artifactId>lombok</artifactId>\n <version>1.18.10</version>\n <scope>provided</scope>\n</dependency>\n\n<dependency>\n <groupId>com.h2database</groupId>\n <artifactId>h2</artifactId>\n</dependency>\n    \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n     <exclusion>\n     <groupId>org.junit.vintage</groupId>\n     <artifactId>junit-vintage-engine</artifactId>\n     </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.batch</groupId>\n <artifactId>spring-batch-test</artifactId>\n <scope>test</scope>\n</dependency>\n\n<dependency>\n <groupId>org.hamcrest</groupId>\n <artifactId>hamcrest-all</artifactId>\n <version>1.3</version>\n <scope>test</scope>\n</dependency>
\n

spring-boot-starter-batch dependency includes all the configurations to run the spring batch application. Lombok is just a helper dependency to write the code faster and cleaner. H2 is used as an in-memory database. spring-boot-starter-test and spring-batch-test are included for test purposes.

\n

Book Class

\n

Let’s create a model class book, which will represent a book. This will just serve as a model class and will help us during implementation of spring batch.

\n
@Data\n\npublic class Book {\nprivate String title;\nprivate String description;\nprivate String author;\n}
\n

Configuration

\n

Let’s create a class called SpringBatchConfiguration. We will add all required configurations here. First, let’s annotate this class with @Configuration to be able to inject beans, and with @EnableBatchProcessing to enable spring batch processing. Additionally, we can add @RequiredArgsConstructor from Lombok which would help us to generate constructor with parameters. These parameters are the ones which are marked as final class properties. These properties will be injected as spring beans. Our class will be like this:

\n
@Configuration\n@EnableBatchProcessing\n@RequiredArgsConstructor\npublic class SpringBatchConfiguration
\n

Now, in SpringBatchConfiguration class let’s add properties which need to be injected into the constructor:

\n
private final JobBuilderFactory jobBuilderFactory;\nprivate final StepBuilderFactory stepBuilderFactory;
\n

jobBuilderFactory and stepBuilderFactory are declared in spring batch jar as spring beans so that we can inject them in any class we want.

\n

File Reader

\n

Now, we need to declare and initialize spring beans to configure the batch process. The first bean which we will need will be responsible for reading from a file line by line. Spring Batch provides a default class for it. The class name is FlatFileItemReader. Similarly, spring has different default reader classes for reading from a relational database, mongodb and etc. However, if you need, you can create your own reader and implement it in a way you want.

\n

Let’s now see what FlatFileItemReader injection will look like.

\n
@Bean\n@StepScope\npublic FlatFileItemReader<Book> bookReader(@Value(\"#{jobParameters['filePath']}\") final String filePath\n) {\nreturn new FlatFileItemReaderBuilder<Book>()\n    .name(\"personItemReader\")\n    .resource(new ClassPathResource(filePath))\n    .delimited()\n    .names(new String[]{\"title\", \"description\", \"author\"})\n    .fieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {{\n    setTargetType(Book.class);\n    }})\n    .build();\n}
\n

We are configuring that the bean reader should read data from the given file path, which should be a CSV file having rows with title, description and author respectively.

\n

The StepScope means to initialize this bean after each step, so file path could be dynamically set when the spring batch job is launched. We will come to that later, how to pass the file path later in this article.

\n

Item Writer

\n

Now, let’s create a writer class, which will take the data and write into the relational database.

\n
@Bean\npublic JdbcBatchItemWriter<Book> writer(final DataSource dataSource) {\n    return new JdbcBatchItemWriterBuilder<Book>()\n        .itemPreparedStatementSetter((book, preparedStatement) -> {\n            preparedStatement.setString(1, book.getTitle());\n            preparedStatement.setString(2, book.getDescription());\n            preparedStatement.setString(3, book.getAuthor());\n        })\n        .sql(\"INSERT INTO books (title, description, author) VALUES (title, description, author_surname)\")\n        .dataSource(dataSource)\n        .build();\n}
\n

In the writer bean, we are setting item processors and adding values inside preparedStatement accordingly, which book property should be inserted for each db table column.

\n

Step Configuration

\n

Now, let’s configure a step which will be executed in the batch process. In our case, the configuration will look like this:

\n
@Bean\npublic Step step1(final ItemWriter<Book> writer, final ItemReader<Book> reader) {\n    return stepBuilderFactory.get(\"step1\")\n        .<Book, Book> chunk(100)\n        .reader(reader)\n        .processor((ItemProcessor<Book, Book>) book -> book)\n        .writer(writer)\n        .build();\n}
\n

In our scenario, we have only one step, but it's also possible to configure multiple steps. Here, we are creating a spring batch step with the name step1 and setting reader and writer accordingly. These are the readers and writers which we created as spring beans earlier.

\n

We are setting chunk as 100, which means the items chunk proceeded is 100. We can make this configurable too. The processor is for converting the current object to the one which the writer should proceed with. In our case, it is the same object.

\n

Job Configuration

\n

Last but not least, let’s configure the job which should be executed.

\n
@Bean\npublic Job importUserJob(final Step step1) {\n    return jobBuilderFactory.get(\"bookReaderJob\")\n        .incrementer(new RunIdIncrementer())\n        .flow(step1)\n        .end()\n        .build();\n}
\n

Here, we create a job with the name bookReaderJob. We add an incrementer RunIdIncrementer, which is an id generator for the tables which are specifically designed to store the details about job executions in the database. After each time the spring batch job is executed, it will save details about the execution. To check the schema structure for storing this data, take a look at this sql. It is for MySql, but other SQL scripts are available too.

\n

Additionally, we add flow by which the job should be executed. Currently, we have only one step in our flow, so we add it.

\n

Please also add this config: spring.batch.job.enabled=false in your properties config file so that spring batch job won’t be executed automatically with no parameters when the application is started.

\n

Execution

\n

To execute the job, we need to declare JobLauncher and launch the job. To do so, we need to create the below class:

\n
@Component\n@RequiredArgsConstructor\npublic class SpringBatchExecutor {  \n    private final JobLauncher jobLauncher;\n    private final Job job;  \n\n    @SneakyThrows\n    public void execute(final String filePath) {\n        JobParameters parameters = new JobParametersBuilder()\n            .addString(\"filePath\", filePath)\n            .toJobParameters();  \n        jobLauncher.run(job, parameters);\n    }  \n}
\n

Now, we can call the execute method from whenever we want, with the file path from which the data should be read.

\n

Additional Classes

\n

There are a few additional classes which you will need to execute your code.

\n

Create a class BookEntity so that spring data JPA will automatically create the book table for you. Then you can create repository.

\n
@Data\n@Entity\n@Table(name = \"books\")\npublic class BookEntity {\n    @Id\n    @GeneratedValue(strategy = GenerationType.IDENTITY)\n    private Long id;\n    private String title;\n    private String description;\n    private String authorFullName;\n}
\n

Then create the interface BookRepository and extend it from JpaRepository. At the moment, we will need this only for testing.

\n
@Repository\npublic interface BookRepository extends JpaRepository<BookEntity, Long> {\n}
\n

Testing

\n

No one likes a code which is not tested. So, let’s write a few test cases for our class.

\n

Here is a code sample for test cases:

\n
@Slf4j\n\n@SpringBootTest\nclass SpringBatchSampleApplicationIntegrationTest {\n\n    @Autowired\n    private SpringBatchExecutor springBatchExecutor;\n    \n    @Autowired\n    private BookRepository bookRepository;\n\n    @Test\n    public void testExecution() {\n        long initialCount = bookRepository.count();\n        assertThat(initialCount, equalTo(0L));\n        springBatchExecutor.execute(\"sample-data.csv\");\n        long count = bookRepository.count();\n        assertThat(count, equalTo(7L));\n    } \n\n    @Test\n    public void testLargeData() {\n        long startTime = System.currentTimeMillis();\n        long initialCount = bookRepository.count();\n        assertThat(initialCount, equalTo(0L));\n        springBatchExecutor.execute(\"large-data.csv\");\n        long count = bookRepository.count();\n        assertThat(count, equalTo(60000L));\n        long endTime = System.currentTimeMillis();\n        log.info(\"executed in miles: {}\", endTime - startTime);\n    }\n}
\n

The second test executes in ≈ 3500 ms in a machine with a 2.2 Ghz Intel Core i7 and 16GB ram. It proceeds 60K lines of CSV file, and saves it into a relational database.

\n

You can check out the working sample code in github,

","fields":{"slug":"/spring-batch-integration/","tags":["auto1","engineering","spring","code","java"]}}},{"node":{"id":"1183b352-9773-54ef-89e7-d09dd84d3a8d","frontmatter":{"category":"Social","title":"Sheep It! Autumn 2019 Hackathon","date":"2019-11-25","summary":"The 2019 autumn edition of AUTO1 Group's hackathon is a wrap, check out the aftermovie and what some of our colleagues developed","thumbnail":{"relativePath":"pages/autumn-hackathon-2019/SheepIt_thumb.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAdABQDASIAAhEBAxEB/8QAGAAAAwEBAAAAAAAAAAAAAAAAAAIEAQP/xAAWAQEBAQAAAAAAAAAAAAAAAAAAAQL/2gAMAwEAAhADEAAAAZcd9KhRJVpRe5QJ/8QAHhAAAgICAgMAAAAAAAAAAAAAAQIAAwQREyISFDP/2gAIAQEAAQUCsd+RrHmMzGvZ5CDMX59hcxMxhpGx+/rSirxX/8QAFxEAAwEAAAAAAAAAAAAAAAAAARARQf/aAAgBAwEBPwEKnF//xAAYEQADAQEAAAAAAAAAAAAAAAAAARECEP/aAAgBAgEBPwF6jGRWvn//xAAhEAACAQIGAwAAAAAAAAAAAAAAAQIhQRESIjFRcTKRof/aAAgBAQAGPwKWuW/JSUvY8W9yWCubj7G43Zlouh9jee/B5/B1uf/EAB8QAAICAgIDAQAAAAAAAAAAAAERACEx8FFxQWGB0f/aAAgBAQABPyENGX5cygn9R4c8z1CELLmUaDJ9QmXdCAKy0XiPlL7hG3QhiEljpxtP2IxfpP/aAAwDAQACAAMAAAAQZ/eB/8QAFxEBAQEBAAAAAAAAAAAAAAAAARARMf/aAAgBAwEBPxAKciRin//EABgRAAMBAQAAAAAAAAAAAAAAAAABESEx/9oACAECAQE/EFIp0WA3p//EAB0QAQADAAMBAQEAAAAAAAAAAAEAESExQVFhgZH/2gAIAQEAAT8QOagIFBrJ+zEm0Ui3Uz1ClAchbykV2Zocj3eN5+zmDWtwHinrAK7/ACBpdpQrNjvXVQfjYh0thW7vcl1Wm81h9+T/2Q==","width":227,"height":325,"src":"/static/dace44993fde07c7486f04f2296f7c35/a2998/SheepIt_thumb.jpg","srcSet":"/static/dace44993fde07c7486f04f2296f7c35/a2998/SheepIt_thumb.jpg 1x,\n/static/dace44993fde07c7486f04f2296f7c35/99cce/SheepIt_thumb.jpg 1.5x,\n/static/dace44993fde07c7486f04f2296f7c35/6e995/SheepIt_thumb.jpg 2x,\n/static/dace44993fde07c7486f04f2296f7c35/968b1/SheepIt_thumb.jpg 3x"}}},"authorName":"Alex Stoicescu","authorDescription":"Alex is Tech Community Manager at AUTO1 Group.","authorAvatar":{"relativePath":"pages/autumn-hackathon-2019/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"

For the second time this year, members of the AUTO1 Group Tech department came together to code, develop exciting projects and have fun! Disclaimer: in an AUTO1 Group hackathon, the focus is not on delivering as many projects as possible - on the contrary, it is about the fun of developing and having joint team sessions in a creative and relaxed setting.

\n

Many thanks to our video crew who did a great job at capturing the atmosphere and collecting impressions, just take a look:

\n

\n

\n \n

\n

The overall theme of the autumn hackathon was SHEEP IT, a pun which should need no explanation. A total of 15 teams competed, with around 40 participants in our Berlin HQ. In addition, there were also colleagues from Kiev, Milan and Katowice that joined remotely.

\n

The winning team, codename “Bike Thieves”, won by developing an anti-theft Slack bot, which ensures that no bicycles can be stolen in the parking space of our Bergmannstraße HQ. When using this Slack bot, you can protect yourself against bicycle theft as you will be informed by the bot and can quickly check and see what’s going on if your bicycle is moved.

\n

As most great ideas, one of our colleagues came up with this during a rather emotionally charged moment for a Berliner - his own bike was stolen during work hours.

\n

There is still a lot that needs to be done before this bot can be used in everyday life. Even under the time constraints of the hackathon, team members Sergey, Rafael and Dmitri made substantial progress on their Smartlock. The team was able to convince the other participants of their idea and their problem-solving strategy, which won them the first place. Some bike owners are already looking forward to using the bot.

\n

Slack bots were a very popular topic at our autumn hackathon, with two other teams doing a language bot and a book exchange bot, while other projects included various internal quality-of-life upgrades to existing workflows, a browser based car game that was rigged to be unbeatable and a somewhat crazy face&object recognition software that was able to tell us, via Slack of course, how many free spots were in a kitchen and how many fruits were there.

\n

In addition to fame and glory, the winners received smart home starter kits, and all participants received a beautifully crafted sheep as a reminder of the hackathon.

\n

What’s next?

\n

The next AUTO1 Group Hackathon will take place in spring 2020 and we look forward to even more exciting projects! Watch this space for updates, wink wink.

","fields":{"slug":"/autumn-hackathon-2019/","tags":["auto1","engineering","social","hackathon","code"]}}}]}},"pageContext":{"slug":"/tags/code","tag":"code","categories":["Architecture","Coding","DevOps","Engineering","ProjectManagement","QA","Social","TechRadar"]}}