{"data":{"allMarkdownRemark":{"edges":[{"node":{"id":"f3a0fb4b-eb2f-5624-a408-d63e3cee5184","frontmatter":{"category":"Coding","title":"Spring batch integration","date":"2020-02-07","summary":"Usage cases for spring-batch.","thumbnail":null,"authorName":"Artur Yolchyan","authorDescription":"Artur is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/spring-batch-integration/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAACXBIWXMAAAsSAAALEgHS3X78AAAEfElEQVQ4yz2O2VMbBQDGlzJQSylXIEc3F0eODYGEJAQI5GBzbK49kt0cm703CSRQqfQCB9taap1qgY61VXqoPZxxbKe2dWo7tY6d0RdHX/0HfPLRF991qTPO/OZ7+33fB+RDFjruKMWhcgwqwnYqaiVhKwXbiolRgZiuVxJ1LrsgEg25sFilFqrFBblQFXNVgajyBMCkxiuIs4yMMoiDSTiwoC3kc/hdVpdVPzFqRua9MpupS7mqmK9LVF0u1CVFzssCIXMYwKZcTGpMyWLUOe2yjoxAQyNO1YCutbV9f1tbe1u7dcRY47GaMiXuVVSFvMTjModLLApwaRefHqeibqNm4EBnb/+A1mo2T9kHcZ9tGR5LQocPvdFmNusEOqv8lPicxBESi4kstifzabeMuj1Wnba7i/Q71vDgJ/Xc4zXhyWrxq1pyG/eWvEMd+9umfWMLAiFWshKD/g8goROV5LjdBJa9ls9r2edr4svTjRenG8836o9WK7e46EXM5zbpNGoVX0qKdEqk00qFgkBnAGrOko+Nz1jA94iZhyvFZ8fZ707JLzaWft48/vJ08/6b5Y+K4azHph5QlbCIXE7yJYRXco8U4Ffvn7ZoAiPgRSp8vU59s9Z89s7RJ2fXtxerbxGpo5nIOWxaCLr1BhOZDUqlOF+ICyVEQSwnAcuBFmtHi8ekRT0OMjBZQyI3lvgduVCLzvGx0HJi9nw+WPTbJ4dNdSLMUTBXiAnF+GsSgKN7n62zxWnQVAIuOjCxEJnaxuevleMfkMjJzPxGcuZTPs4FoA0U3uHIIhHkqKgCr1CIAx6wBeptTY3ZTiRnzmPhbTp9gyFuc+hNBnm6Kj48tXy1Ri7Brs8WxTMVvITN8fkIT8I8GRXIKFCAR11g1wSo3WSIb9dXbojEnUb1C5m5XkJ+ePvI4/WVHSm3BE98SKNCZpbBg6xyntzz92QxN+U39XYBQDMdfbTWvCpiF/Kpyyy1Wy/fapS+XC5tcclq0JGdMOViHhYP/Sdzyn5+HuBR3+xwXwcABG1DNxvkOgqfLxR3q4vvoujFQvzeSuGymKV8w5C+m4p7GTTIYEE2F+b2iABcZnLW2tsNAPrOg5f4zJVyYgfH7vHibY76epm8eyR/BpuU0inPNBFwW6RcSPGVfQYPcbl5gM14Ex69/gDQvW8f7rHuyujtGnm3yT84Id45Sp8lw0dwZHPrqX0s3N/Xnwl56UyAI0KvCQN02lOIOcfV7erWFvBgOwwZj+VSm0LleDlfR+aOsfSDV3+yS1udHV2aAZNaBQ4aByddTjQ6VcGCAJv1wX472K9S96gOHeyZmkHoWJiJ+Jsxz9ax1Ve//33p3k86tVmnM2s0JgWtdlCrHdKDw5AVAoio+7DW2NOj6+lRazQj71/99crayY8b+P3rd3/5459Hv/017o7otENGo82gt5jNo0rq9cMGg0UPWoBZj7O3Dxzo1/f1alyT6OL645Xm2cvXflw+93Rz93tx6YJaZRocHDMYIKMRAg9bQHDYZIIMBrvZ7PwXcPlkRtKKCQsAAAAASUVORK5CYII=","width":50,"height":50,"src":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png","srcSet":"/static/dc9516b9327e627890a1f5c7f84c004a/45876/avatar.png 1x,\n/static/dc9516b9327e627890a1f5c7f84c004a/eb85b/avatar.png 1.5x,\n/static/dc9516b9327e627890a1f5c7f84c004a/4f71c/avatar.png 2x,\n/static/dc9516b9327e627890a1f5c7f84c004a/9ec3e/avatar.png 3x"}}},"headerImage":null},"html":"

Intro

\n

In this article, we will take a look at spring-batch and how it could be used. We will walk through various configurations and we will create an application which reads from a CSV file & writes into a database with outstanding performance.

\n

I used the following: Java 11, Spring 5+, Spring Boot 2 and Maven-based project.

\n

Create a Project

\n

First, we need to create a Spring Boot 2 project. We recommend that you do so by visiting spring initializer, which is a useful tool to generate spring projects with required dependencies and configurations.

\n

Dependencies

\n

We need the dependencies below to run and test the project:

\n
<dependency> \n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter</artifactId>\n</dependency>\n   \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n    <exclusion>\n     <groupId>org.junit.vintage</groupId>\n      <artifactId>junit-vintage-engine</artifactId>\n    </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-batch</artifactId>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-data-jpa</artifactId>\n</dependency>    \n\n<dependency>\n <groupId>org.projectlombok</groupId>\n <artifactId>lombok</artifactId>\n <version>1.18.10</version>\n <scope>provided</scope>\n</dependency>\n\n<dependency>\n <groupId>com.h2database</groupId>\n <artifactId>h2</artifactId>\n</dependency>\n    \n<dependency>\n <groupId>org.springframework.boot</groupId>\n <artifactId>spring-boot-starter-test</artifactId>\n <scope>test</scope>\n <exclusions>\n     <exclusion>\n     <groupId>org.junit.vintage</groupId>\n     <artifactId>junit-vintage-engine</artifactId>\n     </exclusion>\n </exclusions>\n</dependency>\n\n<dependency>\n <groupId>org.springframework.batch</groupId>\n <artifactId>spring-batch-test</artifactId>\n <scope>test</scope>\n</dependency>\n\n<dependency>\n <groupId>org.hamcrest</groupId>\n <artifactId>hamcrest-all</artifactId>\n <version>1.3</version>\n <scope>test</scope>\n</dependency>
\n

spring-boot-starter-batch dependency includes all the configurations to run the spring batch application. Lombok is just a helper dependency to write the code faster and cleaner. H2 is used as an in-memory database. spring-boot-starter-test and spring-batch-test are included for test purposes.

\n

Book Class

\n

Let’s create a model class book, which will represent a book. This will just serve as a model class and will help us during implementation of spring batch.

\n
@Data\n\npublic class Book {\nprivate String title;\nprivate String description;\nprivate String author;\n}
\n

Configuration

\n

Let’s create a class called SpringBatchConfiguration. We will add all required configurations here. First, let’s annotate this class with @Configuration to be able to inject beans, and with @EnableBatchProcessing to enable spring batch processing. Additionally, we can add @RequiredArgsConstructor from Lombok which would help us to generate constructor with parameters. These parameters are the ones which are marked as final class properties. These properties will be injected as spring beans. Our class will be like this:

\n
@Configuration\n@EnableBatchProcessing\n@RequiredArgsConstructor\npublic class SpringBatchConfiguration
\n

Now, in SpringBatchConfiguration class let’s add properties which need to be injected into the constructor:

\n
private final JobBuilderFactory jobBuilderFactory;\nprivate final StepBuilderFactory stepBuilderFactory;
\n

jobBuilderFactory and stepBuilderFactory are declared in spring batch jar as spring beans so that we can inject them in any class we want.

\n

File Reader

\n

Now, we need to declare and initialize spring beans to configure the batch process. The first bean which we will need will be responsible for reading from a file line by line. Spring Batch provides a default class for it. The class name is FlatFileItemReader. Similarly, spring has different default reader classes for reading from a relational database, mongodb and etc. However, if you need, you can create your own reader and implement it in a way you want.

\n

Let’s now see what FlatFileItemReader injection will look like.

\n
@Bean\n@StepScope\npublic FlatFileItemReader<Book> bookReader(@Value(\"#{jobParameters['filePath']}\") final String filePath\n) {\nreturn new FlatFileItemReaderBuilder<Book>()\n    .name(\"personItemReader\")\n    .resource(new ClassPathResource(filePath))\n    .delimited()\n    .names(new String[]{\"title\", \"description\", \"author\"})\n    .fieldSetMapper(new BeanWrapperFieldSetMapper<Book>() {{\n    setTargetType(Book.class);\n    }})\n    .build();\n}
\n

We are configuring that the bean reader should read data from the given file path, which should be a CSV file having rows with title, description and author respectively.

\n

The StepScope means to initialize this bean after each step, so file path could be dynamically set when the spring batch job is launched. We will come to that later, how to pass the file path later in this article.

\n

Item Writer

\n

Now, let’s create a writer class, which will take the data and write into the relational database.

\n
@Bean\npublic JdbcBatchItemWriter<Book> writer(final DataSource dataSource) {\n    return new JdbcBatchItemWriterBuilder<Book>()\n        .itemPreparedStatementSetter((book, preparedStatement) -> {\n            preparedStatement.setString(1, book.getTitle());\n            preparedStatement.setString(2, book.getDescription());\n            preparedStatement.setString(3, book.getAuthor());\n        })\n        .sql(\"INSERT INTO books (title, description, author) VALUES (title, description, author_surname)\")\n        .dataSource(dataSource)\n        .build();\n}
\n

In the writer bean, we are setting item processors and adding values inside preparedStatement accordingly, which book property should be inserted for each db table column.

\n

Step Configuration

\n

Now, let’s configure a step which will be executed in the batch process. In our case, the configuration will look like this:

\n
@Bean\npublic Step step1(final ItemWriter<Book> writer, final ItemReader<Book> reader) {\n    return stepBuilderFactory.get(\"step1\")\n        .<Book, Book> chunk(100)\n        .reader(reader)\n        .processor((ItemProcessor<Book, Book>) book -> book)\n        .writer(writer)\n        .build();\n}
\n

In our scenario, we have only one step, but it's also possible to configure multiple steps. Here, we are creating a spring batch step with the name step1 and setting reader and writer accordingly. These are the readers and writers which we created as spring beans earlier.

\n

We are setting chunk as 100, which means the items chunk proceeded is 100. We can make this configurable too. The processor is for converting the current object to the one which the writer should proceed with. In our case, it is the same object.

\n

Job Configuration

\n

Last but not least, let’s configure the job which should be executed.

\n
@Bean\npublic Job importUserJob(final Step step1) {\n    return jobBuilderFactory.get(\"bookReaderJob\")\n        .incrementer(new RunIdIncrementer())\n        .flow(step1)\n        .end()\n        .build();\n}
\n

Here, we create a job with the name bookReaderJob. We add an incrementer RunIdIncrementer, which is an id generator for the tables which are specifically designed to store the details about job executions in the database. After each time the spring batch job is executed, it will save details about the execution. To check the schema structure for storing this data, take a look at this sql. It is for MySql, but other SQL scripts are available too.

\n

Additionally, we add flow by which the job should be executed. Currently, we have only one step in our flow, so we add it.

\n

Please also add this config: spring.batch.job.enabled=false in your properties config file so that spring batch job won’t be executed automatically with no parameters when the application is started.

\n

Execution

\n

To execute the job, we need to declare JobLauncher and launch the job. To do so, we need to create the below class:

\n
@Component\n@RequiredArgsConstructor\npublic class SpringBatchExecutor {  \n    private final JobLauncher jobLauncher;\n    private final Job job;  \n\n    @SneakyThrows\n    public void execute(final String filePath) {\n        JobParameters parameters = new JobParametersBuilder()\n            .addString(\"filePath\", filePath)\n            .toJobParameters();  \n        jobLauncher.run(job, parameters);\n    }  \n}
\n

Now, we can call the execute method from whenever we want, with the file path from which the data should be read.

\n

Additional Classes

\n

There are a few additional classes which you will need to execute your code.

\n

Create a class BookEntity so that spring data JPA will automatically create the book table for you. Then you can create repository.

\n
@Data\n@Entity\n@Table(name = \"books\")\npublic class BookEntity {\n    @Id\n    @GeneratedValue(strategy = GenerationType.IDENTITY)\n    private Long id;\n    private String title;\n    private String description;\n    private String authorFullName;\n}
\n

Then create the interface BookRepository and extend it from JpaRepository. At the moment, we will need this only for testing.

\n
@Repository\npublic interface BookRepository extends JpaRepository<BookEntity, Long> {\n}
\n

Testing

\n

No one likes a code which is not tested. So, let’s write a few test cases for our class.

\n

Here is a code sample for test cases:

\n
@Slf4j\n\n@SpringBootTest\nclass SpringBatchSampleApplicationIntegrationTest {\n\n    @Autowired\n    private SpringBatchExecutor springBatchExecutor;\n    \n    @Autowired\n    private BookRepository bookRepository;\n\n    @Test\n    public void testExecution() {\n        long initialCount = bookRepository.count();\n        assertThat(initialCount, equalTo(0L));\n        springBatchExecutor.execute(\"sample-data.csv\");\n        long count = bookRepository.count();\n        assertThat(count, equalTo(7L));\n    } \n\n    @Test\n    public void testLargeData() {\n        long startTime = System.currentTimeMillis();\n        long initialCount = bookRepository.count();\n        assertThat(initialCount, equalTo(0L));\n        springBatchExecutor.execute(\"large-data.csv\");\n        long count = bookRepository.count();\n        assertThat(count, equalTo(60000L));\n        long endTime = System.currentTimeMillis();\n        log.info(\"executed in miles: {}\", endTime - startTime);\n    }\n}
\n

The second test executes in ≈ 3500 ms in a machine with a 2.2 Ghz Intel Core i7 and 16GB ram. It proceeds 60K lines of CSV file, and saves it into a relational database.

\n

You can check out the working sample code in github,

","fields":{"slug":"/spring-batch-integration/","tags":["auto1","engineering","spring","code","java"]}}},{"node":{"id":"a3a21c38-873e-56db-a170-eec0e6136040","frontmatter":{"category":"Coding","title":"Writing IntelliJ plugins","date":"2019-12-10","summary":"How to write a simple IntelliJ plugin","thumbnail":{"relativePath":"pages/intellij-plugin/logo.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAAEyElEQVQ4y3WVe0xTVxzHz723rfZBsVeqnc4Yw0aYFSiFFZlZxnSo4w+gSBWEglTmA59jCt4WCOUxq+J4zGwaMqfJfDBi2GbMxjJfMXMzJks0Y5ngfE0mqAwrFLHtPb+de4CKf+wkv5zfOed3Pr/vOb+Te9FAzEokNV9cLvq/Nmpew8L8DIQOANJUPEdh+wGpBB8jrSldo0hd7n0RLJry0fA4bCg2RxYw5S3zm/ISnsWt1vvi8qZMxD1KLkVvb+lmJifS7hrmpJ4lCfidAy8WAqZ8dqzPSwFzIQRN+UEy95SMbxK7MhK3umQiVlnuLVQKwxcVW//SS2O94JNFFZyja3NLHyD03JSHIN5OMxHQZ5C4Fgg0CAlFAIkOAMs6gIWbiV9sQ0cgTlMl+tW7ATSV+E64y2+R9kWVDXJR226PZZSAUvPG5mpxvP0+MfjjjfTg9eh0/Nv8TPGXBdmBS3H5cC52tS8253SffNMN0BZfHdVu7AK+tC9gIIon1MdsvkXvkKojd2aT1N02ZgXDWBnmGBbLJWM5IL4oZzhQEEOMTESsHBBiAjNzz8BrVSJEb721b3taOzdWlIQien+kECfB8gHciM8NkCHOyLLC711d+MPSUkzGIM159njE5IVJ1CfVgXnZbeKbpQ9Ey84+iCy7kxUqylB8QeQoUTeatAWumteIMhKs43m42dMDFotFAgDDMHD+/HlIT88YA7IsNmUcwWmbb4hRu+4BX9mfGQI+SSiY/SBxneeflJLHZ5NKQM7IqKofOzth1qxZMK4QOjo6CDCd+izLwNLlLXhh+R2sr3oEM4SHy8bUOYz07H0daFrQNWPw57XJQG8iIQEwxmC1WiE1NRWampqgu7sbzGYzBUqniLV9gee4B3GE6zHonAMpFAh7ZPSx+sqmH4Pa6XB5++vSHcJbixbhpsZGSEtLA6PRCK2trZCTkxNSK5m+oB2/4h6WYDjcOZgcOvLDHbxyZJfuT6ifDhc3zA8qZIqXNr5sTMiPsJ/CevcoEFgwzPnETGHPBB1VeG+TwQDV2rNnC98FBmlEhUKG1So1TJ06FeRyOchkMuA47iWgzv4t5msCQGB+ldNrDCns3b6AQgGOs42LS67PnR0J+z9pCDY3N+ODhw7i9RvW08qOW8jX2k/jaTUY1M6nI0rhaSSF9e2IRlXvXKD+paJrEbVL9varp6hBGx7m53lejNDxYrhKE6TKGAYj8tiJT4Ea+5mA1o1B5Rz2KwXvnJDClszDtNKfWw9bW7KOwrYl9bAltQGKF3sgb/k+WJpzAGIcXwYiHV/j2Y7vxAjHD4FwxznQurygqQdQunw9yrIBLYV5rEdRQ/YJCty74qujhwp/gt253/R/bP/+eJPj8vvr3JdWvld95X6Spxeiq//Gc2uegMEDoHP7Iaxy5ILKOVREjjtN2k+UIlS34hgF12UeVtRln/i02ta2aWN6q37ydy9mfQdvFK61zqu4CwZnbx8v9Ddrd95PnBxDVI455dknaE9A7OSA+lWn2Orsdi7zo05uYu5V513LTKHXMDHW12Gkcvk4pTBEi6qegLptJ2nfU9CJam1tXKWtjRFWtSND1S06b3H+yiRWXg8l1Ff8y5G3x6oELyJK0cwa6TfwmK79B0KrRX19cxThAAAAAElFTkSuQmCC","width":325,"height":325,"src":"/static/b2f976ed5fa752977d8f8f9b2516534c/b3029/logo.png","srcSet":"/static/b2f976ed5fa752977d8f8f9b2516534c/b3029/logo.png 1x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/8d141/logo.png 1.5x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/ee72c/logo.png 2x,\n/static/b2f976ed5fa752977d8f8f9b2516534c/5dfa8/logo.png 3x"}}},"authorName":"Piotr Czekaj","authorDescription":"Piotr is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/intellij-plugin/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/intellij-plugin/header.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAADGklEQVQozx3R60/bVRjA8V80m2N4IVkWcfpmqyulTdd7oRYopEDbyaXlVgptobANkAzrUpwbyjZvgMpQ7CxhGCaXUTcv04mRRYMkKpB1MLYpIKkymb5S/4WvZ7745DkvnnOey5FS87qQWbqY0zWyIEzrmxkxHOFHcXZkRZCsZ0ixnmKH9TQPWV7m0eIo22q/JmjvI6ENsijyh7I62G7rYXt2F1KgpIo2TZhlbYBFkTCjDaE2Rdhn6mS36TiPC+nml9ilP0a6rQ91wxwZhxJ4ar9kMbuDdeMR+h0DyAOzyBwxpJDhMCsHAlzXBFgQ7mj8DGqaeUr3Agrt8yh0YWQiZolOq9wTuLxXcFZ/SmHNFVrKL/DKwXMcrPkcu2+aYkGKqkKsquu5rvZzQ7gf19R1nFTUo1M0Y5E3YNWGidu6mSzsQdc0j+ZwgrzgLFUVcWEKn/uiMInfPY40pPKxrvWypPFyU7ilqWZJX8dr9l4UrjFUzlFy3OJS8BqWtmVkRzeQtf/K3qNJCsT4EWeUU65+ely9HHOdQ3pR4Wa1OI8Vez637DaWiwp5tfxtIidmGbywxsilJH2xn3lneI3RyXU+iK3Q1votPu9VfP5vOOGMEbeHmXEcoregV+wwvYx4volk816SoSeZrS3lfPmbfFj2OqPeQcbqo4z7+pnwDvBxYJjJ2iFGSt8j5jhLzPUuY7ndXLREmMruZMJyHCmyq4jO3UWMGg1MO+XM5z7LkiHEDwcamVM28JM6wFpODas5Xm7omljRt3DH1Mptcxu/GFv4yNlNZ3CQPk8P14ytSG+l5XHmESund+ZzcmcuX+xx8ldmNckMD78pKthUevjHXMK/5lLuZlYIlWwpq9lSVJJUeQkFL2N77ntsHfO8URpFGk41M572DLFUI++nGDj/sJnb+0q4l1nFH8LW/yrZ2O8mKfdwVxT5/X4x4c8MN+3ip1XtGxj8X+GouYw081gW9+TlfJaiY/xBJWMPKPlkh5bEEwXc3GMnkV7Ad2nZxLepuZpqYHN/GVvisU3h76dLmDKGsTQuoK+7RJNrgP8AU8rtTZmilCkAAAAASUVORK5CYII=","width":1280,"height":720,"src":"/static/6fae10ea7503a11a8f308ac2e2a5217a/26421/header.png","srcSet":"/static/6fae10ea7503a11a8f308ac2e2a5217a/26421/header.png 1x"}}}},"html":"

Background

\n

During analysis of why some services take a long time to build, several problems have been found, mostly related to number of Spring contexts that were created during integration tests. More about those problems can be found in previous blog post. In order to decrease the likelihood that slow tests will appear at AUTO1, we implemented an IntelliJ plugin which warns a developer about possible problems when he writes code and provides quick fixes if they are applicable.

\n

Plugin inspections

\n\n

During the plugin development, we realized that resources on writing IntelliJ plugins are scarce, and therefore decided to create a step by step tutorial on how to implement @DirtiesContext inspection.

\n

Create plugin

\n

In our tutorial we will be using the community edition of IntelliJ 2018.3.5. Final source code is available on github.\nThere are several ways on how to create IntelliJ plugin. The recommended one is to use Intellij plugin for Gradle and that’s what we will use in this tutorial. Let’s start with creating new project using File > New > Project..., select Gradle and make sure that in Additional Libraries and Frameworks both Java and Intellij Platform Plugin are selected.\nIf you cannot see “Gradle” or “Intellij Platform Plugin” then please make sure that both “Gradle” and “Plugin DevKit” IntelliJ plugins are installed.\nContinue next steps of the wizard using default values, use any groupId and artifactId. After some time IntelliJ will download dependencies and create an empty plugin.

\n

Inspection

\n

Our inspection should warn a developer each time there is a class annotated with org.springframework.test.annotation.DirtiesContext and provide quick fix which deletes @DirtiesContext. If Spring is not your thing then any other annotation can be used instead.

\n

There are two kinds of inspections:

\n

Local inspections are executed in the background when file is opened, in general they have access to currently open file. Local inspection cannot report problem for not currently processed file. Inspection class has to extend com.intellij.codeInspection.LocalInspectionTool or one of subclasses.\nGlobal inspections work only in batch mode when analysis is manually triggered via Analyze > Inspect Code and see complete graph of references between classes and can report problem for any file. Inspection class has to extend com.intellij.codeInspection.GlobalInspectionTool or one of subclasses.

\n

We don’t need to access complete graph of references and we would like to give a hint that something is wrong as soon as possible so local inspection is a better choice for our use case. First problem that we encounter is to pick proper base class. One approach is to ask IntelliJ to show class Hierarchy of LocalInspectionTool and take a look what other inspections are extending. In this case AbstractBaseJavaLocalInspectionTool seems to be a good choice since most of Java inspections are based on it.

\n

Create new class named DirtiesContextInspection with following content:

\n
public class DirtiesContextFirstVersionInspection extends AbstractBaseJavaLocalInspectionTool {\n    private static final String DIRTIES_CONTEXT = \"org.springframework.test.annotation.DirtiesContext\";\n\n    private static final String DESCRIPTION_TEMPLATE = \"Usage of @DirtiesContext makes integration tests slower\";\n\n    public String getDisplayName() {\n        return \"Usage of @DirtiesContext is not recommended\";\n    }\n\n    public String getGroupDisplayName() {\n        return GroupNames.PERFORMANCE_GROUP_NAME;\n    }\n\n    public String getShortName() {\n        return \"DirtiesContext\";\n    }\n\n    public boolean isEnabledByDefault() {\n        return true;\n    }\n\n    public PsiElementVisitor buildVisitor(@NotNull final ProblemsHolder holder, \n                                          boolean isOnTheFly) {\n        return new JavaElementVisitor() {\n\n            public void visitAnnotation(PsiAnnotation annotation) {\n                super.visitAnnotation(annotation);\n\n                String qualifiedName = annotation.getQualifiedName();\n\n                if (DIRTIES_CONTEXT.equals(qualifiedName)) {\n                    holder.registerProblem(annotation, DESCRIPTION_TEMPLATE);\n                }\n            }\n        };\n    }\n}
\n

The most interesting thing happens in a visitor which is notified each time a Java annotation is encountered in source code. By overriding different methods, plugin can process methods, classes, imports etc. In our case we check if fully qualified name of annotation matches our expectations and register a problem when that’s the case. Later on we will change this class to include also a quick fix. Creation of inspection class is not enough to make it available in IntelliJ - it’s also needed to register inspection in plugin.xml which among other things describes what plugin does, what other plugins are required and in which version of IntelliJ it can be used.

\n

It’s possible to register each inspection one by one in plugin.xml under extensions tag and configure inspection using xml but we find it easier to register inspectionToolProvider and configure using it in Java code.

\n

Create the following class to implement our inspection provider:

\n
public class CodeInspectionProvider implements InspectionToolProvider {\n    public Class[] getInspectionClasses() {\n        return new Class[]{\n                DirtiesContextInspection.class\n        };\n    }\n}
\n

And register it in plugin.xml:

\n
   <extensions defaultExtensionNs=\"com.intellij\">\n        <inspectionToolProvider implementation=\"com.auto1.intellij.tutorial.CodeInspectionProvider\"/>\n    </extensions>
\n

Now it’s time to check our inspection in action by running runIde gradle task. This could be done from terminal by executing ./gradlew runIde but it’s better to use Gradle view in IntelliJ since it allows to start IDE in debug mode (if needed just right click on task and select Debug, you can create run configuration to speed-up in the future). New IntelliJ instance should show up, if you already have sources of some project that uses Spring Boot then open it, otherwise you could create a new project. Annotate some class with @DirtiesContext annotation and observe inspection marker showing up. In case of problems logs can be found at build/idea-sandbox/system/log/.

\n

Adding quick fix

\n

Many inspections report not only problems but also provide automatic ways of fixing issues. In case of DirtiesContext, it’s not possible to provide safe way of removing it because DirtiesContext is often used when bean holds some state which makes tests dependent on each other. Usually we want to remove the annotation and then figure out the \"dirty parts\" and clean them up in an elegant way. Since the second part is hard to automate we will provide quick fix which only removes annotation.

\n

Go back to DirtiesContextInspection class and add quick fix:

\n
   private static class DeleteQuickFix implements LocalQuickFix {\n        public String getName() {\n            return \"Removes usage of @DirtiesContext\";\n        }\n\n        public void applyFix(@NotNull Project project, @NotNull ProblemDescriptor descriptor) {\n            descriptor.getPsiElement().delete();\n        }\n\n        public String getFamilyName() {\n            return getName();\n        }\n    }
\n

Next step is to pass quick fix when problem is registered:

\n
holder.registerProblem(annotation, DESCRIPTION_TEMPLATE, new DeleteQuickFix());
\n

Executing runIde Gradle task should prove that quick fix works as expected.

\n

Internationalization

\n

Up to this point we have used hardcoded strings inside inspection name and description. To allow the plugin to be accessible in different languages we can externalize the messages. For this purpose we can use properties file. First create PluginBundle class:

\n
public class PluginBundle {\n    private static Reference<ResourceBundle> ourBundle;\n\n    private static final String BUNDLE = \"com.auto1.intellij.tutorial.PluginBundle\";\n\n    private PluginBundle() { }\n\n    public static String message(@NotNull @PropertyKey(resourceBundle = BUNDLE) String key, \n                                 @NotNull Object... params) {\n        return CommonBundle.message(getBundle(), key, params);\n    }\n\n    private static ResourceBundle getBundle() {\n        ResourceBundle bundle = com.intellij.reference.SoftReference.dereference(ourBundle);\n        if (bundle == null) {\n            bundle = ResourceBundle.getBundle(BUNDLE);\n            ourBundle = new SoftReference<>(bundle);\n        }\n        return bundle;\n    }\n}
\n

Then create a new file matching BUNDLE constant, in my case it will be src/main/resources/com/auto1/intellij/tutorial/PluginBundle.properties with content:

\n
inspection.dirties.context.display.name=Usage of @DirtiesContext is not recommended\ninspection.dirties.context.problem.descriptor=Usage of @DirtiesContext makes integration tests slower\ninspection.dirties.context.use.quickfix=Removes usage of @DirtiesContext
\n

Register bundle in plugin.xml:

\n
<resource-bundle>com.auto1.intellij.tutorial.PluginBundle</resource-bundle>
\n

And finally use bundle in inspection, for example:

\n
    public String getDisplayName() {\n        return PluginBundle.message(\"inspection.dirties.context.display.name\");\n    }
\n

Testing

\n

We picked LightPlatformCodeInsightFixtureTestCase as base for our tests because it is recommended in the documentation. Unfortunately, testing appeared harder to set up properly than expected.

\n

First problem was that our tests couldn’t see classes from JDK, which was fixed by specifying project descriptor to use internal JDK:

\n
    protected LightProjectDescriptor getProjectDescriptor() {\n        return new LightProjectDescriptor() {\n            public Sdk getSdk() {\n                return JavaAwareProjectJdkTableImpl.getInstanceEx().getInternalJdk();\n            }\n        };\n    }
\n

Second problem was that visitor received incorrect fully qualified class name of the annotation. Instead of org.springframework.test.annotation.DirtiesContext, it got DirtiesContext while it worked fine for real project in IDE. It turns out that such behaviour occurs when test project doesn’t see definition of some class. This is fixable by either hardcoding problematic class into test or by adding dependency as library to the project. Second approach avoids copying source code from other projects and seems to be more interesting so it will be presented here. In order to download dependency jar we use ShrinkWrap library:

\n
private File[] getMavenArtifacts(String... mavenArtifacts) {\n        File[] files = Maven.resolver()\n                            .resolve(mavenArtifacts)\n                            .withoutTransitivity()\n                            .asFile();\n        if (files.length == 0) {\n            throw new IllegalArgumentException(\"Failed to resolve artifacts \" + Arrays.toString(mavenArtifacts));\n        }\n        return files;\n    }
\n

When dependency is resolved and downloaded into local Maven cache it can be added as library with code listed below:

\n
   protected void attachMavenLibrary(String mavenArtifact) {\n        File[] jars = getMavenArtifacts(mavenArtifact);\n        Arrays.stream(jars).forEach(jar -> {\n            String name = jar.getName();\n            PsiTestUtil.addLibrary(myFixture.getProjectDisposable(), myModule, name, jar.getParent(), name);\n        });\n    }
\n

It’s important to use myFixture.getProjectDisposable() instead of myFixture.getProject() otherwise there is an exception during test shutdown:

\n
com.intellij.openapi.util.TraceableDisposable$DisposalException: Virtual pointer 'jar:///somePath/.m2/repository/org/springframework/spring-test/5.1.5.RELEASE/spring-test-5.1.5.RELEASE.jar!/' hasn't been disposed
\n

Next surprise is that by default test searches for test data in strange location inside IntelliJ home folder which can be fixed with overriding getTestDataPath. Since input files most likely won’t compile because of missing imports and possible usage of special markers like we don’t use src/test/java folder to store them:

\n
    @Override\n    protected String getTestDataPath() {\n        return \"src/test/testData\";\n    }
\n

Inspection tests can be done by providing file to analyse and resulting file that should be created after given quick fix has been applied. Inspection tests use configureByFile to load input file, doHighlighting to trigger source code analysis, launchAction to execute quick fix and finally checkResultByFile to compare results against after file.

\n
    myFixture.configureByFile(testName + \".java\");\n    myFixture.enableInspections(new DirtiesContextInspection());\n    myFixture.doHighlighting();\n    IntentionAction quickFixAction = myFixture.findSingleIntention(intentionHint);\n    myFixture.launchAction(quickFixAction);\n    myFixture.checkResultByFile(testName + \".after.java\");
\n

Complete source code can be found at github github

\n

Next steps

\n

If you would like to share your plugin with other developers, you can publish to JetBrains plugin repository as described in the documentation. Other simple option is to execute buildPlugin Gradle task which will create plugin zip file inside build/distributions and then install it via Install Plugin from disk... available inside Preferences > Plugins (in IntelliJ 2018.3 is available through \"gears icon\").

\n

When working on your own ideas you might run into a situation when you don't know how to implement some functionality. In this situation you could try to find the answer using links provided in section below. What also worked for us was reading source code of inspections available as part of community edition of IntelliJ, often there is an existing inspection which does a similar thing to what you might want to do.

\n

Links

\n","fields":{"slug":"/intellij-plugin/","tags":["intellij"]}}},{"node":{"id":"df094608-f958-5bcd-b406-ea359067fbd8","frontmatter":{"category":"Coding","title":"PostgreSQL 12 - a precious release","date":"2019-10-29","summary":"Improved Common Table Expressions in recent release of Postgres","thumbnail":{"relativePath":"pages/postgres12-a-precious-release/elephant_cropped.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAWABQDASIAAhEBAxEB/8QAGQABAQADAQAAAAAAAAAAAAAAAAMBAgQF/8QAFgEBAQEAAAAAAAAAAAAAAAAAAQIA/9oADAMBAAIQAxAAAAHuzwyh9NJUx1M1Cf/EABoQAAIDAQEAAAAAAAAAAAAAAAEDABESAhD/2gAIAQEAAQUC9ZwwnDAsaq5ZgM//xAAYEQADAQEAAAAAAAAAAAAAAAAAARICE//aAAgBAwEBPwGDmxaZbP/EABcRAAMBAAAAAAAAAAAAAAAAAAACEwH/2gAIAQIBAT8BoVU1MJqf/8QAGxAAAgEFAAAAAAAAAAAAAAAAADEBECAhQWH/2gAIAQEABj8CrPIwNM1Z/8QAHBABAAICAwEAAAAAAAAAAAAAAQARITEQQVFx/9oACAEBAAE/IVDbLPeEWs5BNbatDthqyfsUeMAiBon/2gAMAwEAAgADAAAAEL/Xfv/EABkRAQEAAwEAAAAAAAAAAAAAAAEAMUFRYf/aAAgBAwEBPxAOy4kkzet//8QAGREAAwADAAAAAAAAAAAAAAAAAAExEVFh/9oACAECAQE/EHnGb2IuHE//xAAdEAEAAgICAwAAAAAAAAAAAAABABEhMRBRYXGh/9oACAEBAAE/EFAAVogmhwCxZXVnre/cLABvFBlrzTMCB1e/kSbKO4RYmd3m4ZIf/9k=","width":292,"height":325,"src":"/static/5076cf686ef8d2a21a3a6da9bf016970/a2998/elephant_cropped.jpg","srcSet":"/static/5076cf686ef8d2a21a3a6da9bf016970/a2998/elephant_cropped.jpg 1x,\n/static/5076cf686ef8d2a21a3a6da9bf016970/99cce/elephant_cropped.jpg 1.5x,\n/static/5076cf686ef8d2a21a3a6da9bf016970/6e995/elephant_cropped.jpg 2x,\n/static/5076cf686ef8d2a21a3a6da9bf016970/968b1/elephant_cropped.jpg 3x"}}},"authorName":"Mariusz Nowak","authorDescription":"Mariusz is a Senior Software Engineer at AUTO1 Group.","authorAvatar":null,"headerImage":null},"html":"

The PostgreSQL team announced recently a new release of the most advanced\nopen source relational database - PostgreSQL 12. As usual it comes with an impressive list of improvements\n(generated columns ♥️), one of them being long awaited by dozens of developers: improved Common Table Expressions.

\n

I should first explain what are the Common Table Expressions for those who are unfamiliar with them:\nthe CTE’s, often called “WITH queries”, are SQL constructs giving a possibility of creating temporal data views for a sake of a query execution.

\n

Essentially CTE is an additional query which results can be referenced in the subsequent CTE’s or the main query before which\nit is being placed. It should be clear enough with an example - here’s a sample query with two CTE’s taken from Postgres docs:

\n
WITH regional_sales AS (\n    SELECT region, SUM(amount) AS total_sales\n    FROM orders\n    GROUP BY region\n), top_regions AS (\n    SELECT region\n    FROM regional_sales\n    WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)\n)\nSELECT region,\n       product,\n       SUM(quantity) AS product_units,\n       SUM(amount) AS product_sales\nFROM orders\nWHERE region IN (SELECT region FROM top_regions)\nGROUP BY region, product;
\n

Now we know what they are, but what purpose can they serve us? Well - we could parry here and say: for the same purpose as ordinary database views serve.\nThat’s of course a dramatic simplification - CTE’s are much more powerful and should not be treated as a simple database views.\nNevertheless I would like to keep the collation for the sake of this article.

\n

Database views are absolutely optional - one can simply substitute them with a subquery and achieve identical results.\nIndeed that’s what modern database engines do these days - once a database view is being used they inline it’s query as a subquery.\nWhy to bother then? We are able to deal completely without database views and even if we did use one the database engine would get rid of it anyway.

\n

What benefits do views give us then? And why the heck are they inlined?

\n

The beauty of database views

\n

The most appropriate explanation here is that database views help us achieve better readability.\nThey offer an elegant way of abstracting some parts of a database into a meaningful object, often matching closely with the domain.

\n

Instead of creating giant and ugly looking queries it is possible to extract some of its parts into an appealing view which is easier to browse and select data from. \"Divide and conquer\" rule in it’s true form.

\n

Still, we didn’t answer the fact that the underlying view’s query is most of the times inlined while it is being referenced. The reason is performance of course. Smart guys found out that lazy evaluation helps the optimizer a lot - by delaying the execution we could take advantage of a context of the actual query.

\n

This in turn allows many clever optimization techniques, like: pushing down predicates (WHERE filters), eliminating unnecessary JOINS, accessing only subset of columns etc.\nIn other words - a database is smart enough to do as little work as possible when evaluating database views, in the context of the issued query.

\n

Personally I love this pattern: aggregating all the data into views and letting a database to optimize my queries - these folks are really good in it and my queries are dead simple too.

\n

I have mentioned the CTE’s at the beginning, saying they are able to create temporal data views. I still conform to the comparison with database views - they both are in many cases similar.\nThe main difference is that CTE results are temporary and are reachable only in the context of a query which CTE is being part of.

\n

It may make sense to use a CTE in a place where a regular database view is not justified (e.g. it makes sense only in the context of a query and not in the whole domain), expecting similar behavior.

\n

An ugly brother

\n

Besides many remarkable advantages there’s at least one disadvantage which disqualifies CTE's for most use cases - before PostgreSQL 12 it was implemented as an optimization fence. What does it mean?

\n

Easy to imagine an example with a generic data view aggregating lots of data. If the view is then used to select just few records it could mean a tremendous waste of computation, if the aggregation is executed immediately.\nInstead the aggregation should be executed only on a small subset of data, which can be deduced from the outer query.

\n

Unfortunately such counter intuitive behavior was true for CTE’s for a long time - their results were materialized only to be accessible for the rest of the query afterwards.

\n

Not to blame anybody - the creators had quite good reasons to implement such behavior (i.e. guarantee of exactly one evaluation, possibility of a recursive CTE’s and more) but this still feels like focusing on corner cases instead of optimizing the happy path.\nThat’s why the community insisted for a long time for changing the status quo by giving the possibility of disabling the fence and unlocking the full potential of CTE’s.

\n

Game changer

\n

The SQL gods listen to their prayers and here it is - PostgreSQL 12 with updated Common Table Expressions.\nBy default, when few constraints are met, the queries will be inlined allowing joint optimizations.

\n

It is still possible to force the old behavior - by defining the CTE AS MATERIALIZED the engine would execute it immediately.\nIt is also possible to hint the optimizer that we definitely want the CTE to be inlined, e.g. when the CTE is being referenced twice the engine won't inline it by default.

\n

This is truly a game changer for many developers who care about their queries’ readability, allowing them to substitute not very well liked subqueries with elegant “WITH queries”.

\n

Don’t get me wrong - not every subquery should be immediately replaced, they still have their strengths and in some cases they should be chosen over CTE’s. It is just convenient to have two distinct tools in a toolbox, isn’t it?

","fields":{"slug":"/postgres12-a-precious-release/","tags":["postgres","postgres12","release","sql","cte"]}}},{"node":{"id":"24fd2709-272a-5811-a69c-7100924de21c","frontmatter":{"category":"Coding","title":"Serving JSON (almost) directly from database","date":"2019-10-29","summary":"Learn how to query JSON data from Postgres 12 to improve Java application's performance.","thumbnail":{"relativePath":"pages/serving-json-from-postgres/liberty.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAIDBAX/xAAUAQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAHryz0HUYxXEAP/xAAbEAABBAMAAAAAAAAAAAAAAAABAgMSIhAgIf/aAAgBAQABBQJw0SoKxLjVDPT/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/AR//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/AR//xAAYEAACAwAAAAAAAAAAAAAAAAABIQAQIP/aAAgBAQAGPwJRGzn/xAAaEAEAAwADAAAAAAAAAAAAAAABABEhEDGB/9oACAEBAAE/IVEqq3yWWlcbozCOmOokIz//2gAMAwEAAgADAAAAEOAPAP/EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8QH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8QH//EABwQAQADAQADAQAAAAAAAAAAAAEAETEhQXGhwf/aAAgBAQABPxBIyYq1CL8uc5XU14YIGwTYJTDeFrX2G/k7OwjIAbIBek//2Q==","width":325,"height":325,"src":"/static/e0de7302edcea3d0b0c19a9d147b547a/a2998/liberty.jpg","srcSet":"/static/e0de7302edcea3d0b0c19a9d147b547a/a2998/liberty.jpg 1x,\n/static/e0de7302edcea3d0b0c19a9d147b547a/99cce/liberty.jpg 1.5x,\n/static/e0de7302edcea3d0b0c19a9d147b547a/6e995/liberty.jpg 2x,\n/static/e0de7302edcea3d0b0c19a9d147b547a/968b1/liberty.jpg 3x"}}},"authorName":"Mariusz Nowak","authorDescription":"Mariusz is a Senior Software Engineer at AUTO1 Group.","authorAvatar":null,"headerImage":null},"html":"

RESTful API’s are ruling the industry nowadays. They are very often associated with an extremely popular data format - JSON.\nIt is so popular that database creators decided to implement it directly in the database engines.\nAll modern databases offer JSON processing capabilities.\nLet’s have a look and see how can we make use of them to improve our application’s performance.

\n

JPA or JDBC?

\n

The typical choice for a Java application is to use JPA as the persistence layer abstraction.\nIt became the de-facto standard for a reason - it offers great flexibility and ease of use.\nUnfortunately it doesn't come without a price since we have to pay with degraded performance.

\n

It is a widely known fact that ORM’s impact performance, compared to pure JDBC.\nOn the other hand, relatively few are willing to use plain JDBC, considering the amount of additional work it requires.

\n

Let me demonstrate how to use JDBC to leverage the performance, with the help of modern Postgres features.

\n

Demo time

\n

I have prepared a demo application running on Spring Boot which exposes example data from the northwind database.\nThe data are exposed in three different ways.

\n

The baseline version simply exposes JPA entities:

\n
@GetMapping(path = \"/v1/orders/{order_id}\",\n            produces = APPLICATION_JSON_VALUE)\npublic Order getEntity(@PathVariable(\"order_id\") Integer orderId) {\n    return orderRepository.findById(orderId).orElseThrow();\n}
\n

The Order entity aggregates data from numerous database tables: orders, order_details, products, customers, customer_customer_demo and customer_demographics.\nYou could check the corresponding JPA mapping here.

\n

Thanks to the magic of Spring the entity is being translated into a JSON response:

\n
{\n  \"orderId\": 10284,\n  \"employeeId\": 4,\n  \"orderDate\": \"1996-08-19\",\n  \"requiredDate\": \"1996-09-16\",\n  \"shippedDate\": \"1996-08-27\",\n  \"shipVia\": 1,\n  \"freight\": 76.56,\n  \"shipName\": \"Lehmanns Marktstand\",\n  \"shipAddress\": \"Magazinweg 7\",\n  \"shipCity\": \"Frankfurt a.M.\",\n  \"shipRegion\": null,\n  \"shipPostalCode\": \"60528\",\n  \"shipCountry\": \"Germany\",\n  \"orderDetails\": [\n    {\n      \"orderId\": 10284,\n      \"productId\": 27,\n      \"product\": {\n        \"productId\": 27,\n        \"categoryId\": 3,\n        \"supplierId\": 11,\n        \"productName\": \"Schoggi Schokolade\",\n        \"discontinued\": false,\n        \"quantityPerUnit\": \"100 - 100 g pieces\",\n        \"unitPrice\": 43.9,\n        \"unitsInStock\": 49,\n        \"unitsOnOrder\": 0,\n        \"reorderLevel\": 30\n      },\n      \"unitPrice\": 35.1,\n      \"quantity\": 15,\n      \"discount\": 0.25\n    },\n    {...},\n    {...},\n    {...}\n  ],\n  \"customer\": {\n    \"customerId\": \"LEHMS\",\n    \"companyName\": \"Lehmanns Marktstand\",\n    \"contactName\": \"Renate Messner\",\n    \"contactTitle\": \"Sales Representative\",\n    \"address\": \"Magazinweg 7\",\n    \"city\": \"Frankfurt a.M.\",\n    \"region\": null,\n    \"postalCode\": \"60528\",\n    \"country\": \"Germany\",\n    \"phone\": \"069-0245984\",\n    \"fax\": \"069-0245874\",\n    \"customerDesc\": [\n      {\n        \"customerTypeId\": \"AAAA\",\n        \"customerDesc\": \"First demographic\"\n      },\n      {...},\n      {...},\n      {...}\n    ]\n  }\n}
\n

The demo project includes the p6spy library which gives an insight into the queries being issued against database.\nTurns out that three queries are required to load a particular Order - this means three database round trips.

\n

Of course we can force the ORM to join all the tables and fetch them in one hop, but this seems a not so elegant solution -\nit pushes the ORM to load duplicated data and the necessity of deduplicating them in-memory. Can we do better?

\n

JDBC to the rescue

\n

JDBC is always a choice if the solution could be expressed via SQL query. Luckily we can create one which matches our case perfectly:

\n
WITH products(\"productId\", \"categoryId\", \"supplierId\", \"productName\", \"discontinued\", \"quantityPerUnit\",\n              \"unitPrice\", \"unitsInStock\", \"unitsOnOrder\", \"reorderLevel\") AS\n         (SELECT p.product_id,\n                 p.category_id,\n                 p.supplier_id,\n                 p.product_name,\n                 p.discontinued::boolean,\n                 p.quantity_per_unit,\n                 p.unit_price,\n                 p.units_in_stock,\n                 p.units_on_order,\n                 p.reorder_level\n          FROM products p),\n     orderDetails(\"orderId\", \"productId\", \"product\", \"unitPrice\", \"quantity\", \"discount\") AS\n         (SELECT d.order_id,\n                 d.product_id,\n                 (SELECT to_json(p) FROM products p WHERE p.\"productId\" = d.product_id),\n                 d.unit_price,\n                 d.quantity,\n                 d.discount\n          FROM order_details d),\n\n     demographics(\"customerId\", \"customerTypeId\", \"customerDesc\") AS\n         (SELECT ccd.customer_id,\n                 ccd.customer_type_id,\n                 cd.customer_desc\n          FROM customer_demographics cd\n                   JOIN customer_customer_demo ccd on cd.customer_type_id = ccd.customer_type_id),\n     customers(\"customerId\", \"companyName\", \"contactName\", \"contactTitle\", \"address\", \"city\", \"region\", \"postalCode\",\n               \"country\", \"phone\", \"fax\", \"customerDesc\") AS\n         (SELECT c.customer_id,\n                 c.company_name,\n                 c.contact_name,\n                 c.contact_title,\n                 c.address,\n                 c.city,\n                 c.region,\n                 c.postal_code,\n                 c.country,\n                 c.phone,\n                 c.fax,\n                 (SELECT to_json(array_agg(to_json(d))) FROM (SELECT \"customerTypeId\", \"customerDesc\" FROM demographics WHERE demographics.\"customerId\" = c.customer_id) d)\n          FROM customers c),\n\n     orders(\"orderId\", \"employeeId\", \"orderDate\", \"requiredDate\", \"shippedDate\", \"shipVia\", \"freight\",\n            \"shipName\", \"shipAddress\", \"shipCity\", \"shipRegion\", \"shipPostalCode\", \"shipCountry\", \"orderDetails\",\n            \"customer\") AS\n         (SELECT o.order_id,\n                 o.employee_id,\n                 o.order_date,\n                 o.required_date,\n                 o.shipped_date,\n                 o.ship_via,\n                 o.freight,\n                 o.ship_name,\n                 o.ship_address,\n                 o.ship_city,\n                 o.ship_region,\n                 o.ship_postal_code,\n                 o.ship_country,\n                 (SELECT to_json(array_agg(to_json(d))) FROM orderDetails d WHERE d.\"orderId\" = o.order_id),\n                 (SELECT to_json(c) FROM customers c WHERE c.\"customerId\" = o.customer_id)\n          FROM orders o)\n\nSELECT to_json(orders) AS json_body FROM orders WHERE orders.\"orderId\" = ?
\n

The query looks verbose, perhaps a little enigmatic.\nIt is a bit long and therefore it could be overwhelming.\nHowever, on a closer look we could notice it isn't so much complicated - there are few idioms which make the query powerful.

\n

The good news is that the query generates the exact same JSON as the JPA endpoint provides - it is possible thanks to Postgres’ excellent support for JSON format.\nOur case is not sophisticated though - single to_json function is enough to deal with JSON objects and arrays.

\n

One can notice how the nesting has been realized: how to nest JSON objects and JSON arrays, e.g. the orderDetails or the customer into order).\nThe simple idiom (accordingly to_json(...) or to_json(array_agg(to_json(...)))) does the job.

\n

The database query could be adapted according to the caller needs - the response can be shrinked, making the query simpler and - most probably - even more effective.

\n

Worth noticing is that the join table customer_customer_demo has been mitigated in the end result, just like the JPA version does by using @ManyToMany.

\n
\n
\n

The query is flexible enough to substitute all JPA features we need.

\n
\n
\n

I want also emphasize the fact of CTE’s being used to decompose the query into logical parts.\nPersonally I find it elegant because it is easy on the eyes and it “scales” - meaning while our case gets more complicated (JSON aggregates more data, becomes deeply nested) our query gets longer instead of getting deeper.\nIf the query grows too long we can simplify it by extracting some it’s parts as database views.

\n

The CTE has one more advantage here: it gives a control over the keys visible in end resulting JSON (order_id becomes orderId).

\n

So we have a nice query which allows us to fetch required data in one hop in a format ready to be consumed by our clients.\nThis means no adjustments are required, therefore the Java implementation can be that easy:

\n
@SneakyThrows\n@GetMapping(path = \"/v2/orders/{order_id}\",\n            produces = APPLICATION_JSON_VALUE)\npublic String getRawBodyFromDatabase(\n            @PathVariable(\"order_id\") Integer orderId) {\n    return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n        rs.next();\n        return rs.getString(1);\n    }, orderId);\n}
\n

We didn’t use plain JDBC here but a JdbcTemplate which has a negligible overhead and it simplifies our code.

\n

Of course we could achieve similar results using plain JDBC (e.g. by using PreparedStatement directly) - without the use of Spring.\nEven then the necessity of extracting only a single field from the result set makes the solution less coarse than using JDBC in a usual way\n(in our case it would require managing at least three result sets).\nAll the complexity has been moved down into the query.

\n

But I need a POJO!

\n

Some folks could argue “I don’t want the JSON to be directly returned - I definitely need the POJO to make some processing over it”.\nThat's an absolutely fair statement! We can satisfy this requirement as well.

\n

Let’s point it one more time: our query produces the exact same JSON as the Order JPA entity has been translated into.

\n

This means we can use the resulting JSON and unmarshall it into an object of class Order.\nOf course the object won’t be tracked by the JPA whatsoever but we don’t need these capabilities - remember: we are serving persisted data and not mutating them.

\n
@SneakyThrows\n@GetMapping(path = \"/v3/orders/{order_id}\",\n            produces = APPLICATION_JSON_VALUE)\npublic Order getEntityNotUsingJpa(@PathVariable(\"order_id\") Integer orderId) {\n    return jdbcTemplate.query(JSON_QUERY, (ResultSet rs) -> {\n        rs.next();\n        return unmarshall(rs.getString(1), Order.class);\n    }, orderId);\n}\n\n@SneakyThrows\nprivate <T> T unmarshall(String json_body, Class<T> clazz) {\n    return objectMapper.readValue(json_body, clazz);\n}
\n

This version is very similar to V2 but it introduces an additional step of unmarshalling.\nIt is also a great verification step which makes us sure the V1 and V2 results are identical.

\n

Measure, measure, measure

\n

So how are all the solutions performing?\nWe suspect the V2 version to be faster than the competitors.\nWe are also unsure about the V3 which requires additional unmarshalling phase - could it impact the performance badly?

\n

The demo project includes a JMeter configuration and a bash script which verifies the configurations.\nThe benchmark is always run against a fresh setup (db + app).\nThe app is running on java 11 with EpsilonGC turned on - this allows us to verify memory consumption of each solution.

\n

The requests are run in strictly defined sequence, fetching the orders one by one.\nThere’s a prewarming phase which issues 100 requests and then a profiling phase with 20k of requests.

\n

I have run the benchmark several times on a Macbook Pro 2018, the results are similar on each run.\nHere’s a summary of the most recent run:

\n\n

We can see the V2 outperforming the V1 by 32 seconds - that’s 37% improvement.

\n

\n \n \n \n \n

\n

We can notice also a lower memory consumption on the V2 - it used 52 MB (49 MiB) less memory than V1\n(noted the initial app’s memory usage which is 785 MB):

\n

\n \n \n \n \n

\n
\n
\n

Worth noticing: the V2 version does not leverage Hibernate at all therefore we could minimize the memory footprint even more by excluding the JPA dependencies.

\n
\n
\n

Let’s have a look at response times:

\n

\n \n \n \n \n

\n

Once again we are able to notice improved throughput as well as lower latency.

\n

The plot confirms that as well (cropped for brevity):

\n

\n \n \n \n \n

\n

Personally I am very well satisfied with the V3 results - the memory consumption being at the same level as the JPA it still offers satisfying performance, nearly as good as the V1.\nIt proves that the unmarshalling does not impact the performance as badly as we might expect.

\n

I really encourage readers try running the benchmark and share their own results!

\n

Postgres 12 exclusive

\n

Last words about the database: the results are reproducible currently only on Postgres 12,\ndue to recent improvements in Common Table Expressions.

\n

I have had a run on Postgres 11 and the V2’s performance was… horrible - I had to shorten the benchmark 10 times to make it finish in a reasonable time.\nHere are the results I got:

\n

\n \n \n \n \n

\n

You can give it a try - the demo app is prepared to allow a quick switch to Postgres 11.

\n

Summary

\n

The query we have prepared enables fine control over the data being fetched from database.

\n

We are no longer restricted to tabular data - we could easily fetch complex data graphs, even manipulate them to match the desired format.\nWe did it keeping the solution maintainable and very effective in terms of throughput.

\n

It is still unknown to me what is the main source of latency in the JPA version: either is it the additional round trips or the internals of Hibernate.\nI see the topic interesting to explore.\nPerhaps some day we’ll get an ORM which is able to load entity graphs using a single query.\nAny Hibernate devs reading here?

","fields":{"slug":"/serving-json-from-postgres/","tags":["jdbc","jpa","postgres12","json","spring boot"]}}},{"node":{"id":"a7cdd55d-a270-5c8c-be42-5c2c15c043f0","frontmatter":{"category":"Coding","title":"Testing your isolated Fragments with Koin","date":"2019-09-09","summary":"This blog post explains how you could set up your environment in order to test your fragments in isolation from other parts of your application","thumbnail":{"relativePath":"pages/testing-isolated-fragments-with-koin/thumbnail.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsTAAALEwEAmpwYAAAEcklEQVQ4y5WUbUxTVxjHL2UxCouJbsk+7MvmErNsH7b4YX4wsg0nU6RbXEXKS3lxMCm1paGIaJ1ji04Z08Eob8MhaGCSwRZIkPEqVSgKZVJKLb194basyHiZa0tfGG2fPfde6sQsLjvJP885zznP7z7n3PMcYoksIZZNcoJuDuPZsBWzkEM81uA9gnCNH2N8MEgQDzWniKc25xS7wEEWhYV8S+aarQ7Tl286jOdfCvmW7whY6Nq4IjeRUEgSmX7IPmoO8gsG9rupIdJtkpd5TPmLXlK66tFLvMuTuSrH5Mld9Ly6jhuukKZxSnPiwy5kcNfASethf0ydZ2BWfctGt7HgNiwVwl+kCLy6o+CdyAK4nwm+sQxwjYl4T+4OwREh6JnD77JOnyEnnNm6vvBzmD8O3slPfB5NZsAzlh703E0OBEeTYGX0CFj7Zd+KuanPVUr5u74R8raVCg9t/+54+st0bFWeIOxc6j7iTPw7LPTexPgGlzaXDNzPAgQF3HdTgu4hftAzlAyLt4W3dL3F0ReFSdcVEv4qnpe/XMz3XcrmzRXx9zCEc6n7mfP9NH4tS7uufqtzTLjsG00GGuRW8vx+FR9mbxVU0fNfZ8W3VObydQgzVeYmAdpV3GZf8RHugpwXtYNe81lC9D+3Y3yiN+JPVaZ9ZegQuPoPBlxd3KC78wC4b8TOUs281rqTaXApJ3FBIU6AyzIB1OalgELMX7ySLwA8u19oRnFGXBiTIXRvZ8jzStE1GOaBqyPW52zbF3T+tDfoad0Dwba9MN/4PnReiH+gkH3cUJ2fVl+dn06rsUqW1nQ2dX8Rc50AiE10h7p5mgFODtZuW+xMXoTOGHA2v+13NkX5HVd3+53fvxVYbtgJv6kbneaZ+UkLRekslE1nsdq0ZspqGNdO9nR190TSDM3EBHuX7T1sJej6Lr4x15425mqJhZUfY8DdFA3z1z/ymYabA6TZBlZqGiiKYjU9DTarFUZGRpZ1Ot2NmZmZLew9vClg7IPetZLD1PU9X0VTHflZ5nZpnFar3m0wWb1GkgSDwRAIiSTJldnZWVCpVFKTySRHH1u/+r4Swt3O/m5bdx7n38qTNOi77XY70BBUkAYajcagVquFgYGB17BfjT7ho4B7vXXEw4549gPdJYSpU86xtYvCzYP1G2gfBuzELJitYqAfx2Cz2UCtVlN6vX4Ax7UajYbz1EcDAxk7NTXFLMTMeOhbslgsQMMR0IDZPY/AF0MxSqUybB3k8VcD+xxaZUIep+1y6TPMXxwbfcGg13+g007sIP5PKzuW8J9rrgxbnr2qWdhU9+tcRM3PvZtr1HORla1dG9ctol+NMlFCKMNorF1uuSQxhu6jPYzzUZUFmSkVhdlCRWF2uuLEUQEqQyEXf6g4kS2rOC15dR2wXJzInIFCnPgKQjRYXo1oe/AxaEJ4P/bvoK8KS1CJ+oFWOWs7Uf2ohCeA/JDdjMH5CI5DewrVgRKh/wBm+joqD8cHURKUEMfZaGUV0uQtfwP/KbEPkxHssgAAAABJRU5ErkJggg==","width":280,"height":280,"src":"/static/3002cc6db63b27d120e79145783a1750/b2831/thumbnail.png","srcSet":"/static/3002cc6db63b27d120e79145783a1750/b2831/thumbnail.png 1x"}}},"authorName":"Oleg Osipenko","authorDescription":"Senior Software Engineer","authorAvatar":{"relativePath":"pages/testing-isolated-fragments-with-koin/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAQDAf/EABYBAQEBAAAAAAAAAAAAAAAAAAIAA//aAAwDAQACEAMQAAABvxphrZG0EPTN4Bn/xAAcEAABBQADAAAAAAAAAAAAAAACAAEDERIEIUH/2gAIAQEAAQUCMe3HKpTU69LkszxyFqUy1a//xAAWEQADAAAAAAAAAAAAAAAAAAABIDH/2gAIAQMBAT8BET//xAAXEQADAQAAAAAAAAAAAAAAAAAAAREQ/9oACAECAQE/AW3S7//EABkQAQADAQEAAAAAAAAAAAAAAAEAEDERcf/aAAgBAQAGPwIJtFZFXvs2v//EABoQAQEBAQEBAQAAAAAAAAAAAAEAESExUWH/2gAIAQEAAT8hxA4Prax0i9GTYNSGfefshNWsqhiiOSvft//aAAwDAQACAAMAAAAQpBd//8QAGBEBAQADAAAAAAAAAAAAAAAAAQAQEUH/2gAIAQMBAT8QAzTcx//EABURAQEAAAAAAAAAAAAAAAAAABBB/9oACAECAQE/EAVP/8QAHBABAQEBAAIDAAAAAAAAAAAAAREAITFBgZHR/9oACAEBAAE/EGgFUPPPR96QjAi/JqtWdBTjHT0kCCvN/MUGjjrFKJRrNDGeA876ympRXm//2Q==","width":50,"height":50,"src":"/static/4d1dd08092290e44fd34c92f78d9904e/d2d31/avatar.jpg","srcSet":"/static/4d1dd08092290e44fd34c92f78d9904e/d2d31/avatar.jpg 1x,\n/static/4d1dd08092290e44fd34c92f78d9904e/0b804/avatar.jpg 1.5x,\n/static/4d1dd08092290e44fd34c92f78d9904e/753c3/avatar.jpg 2x,\n/static/4d1dd08092290e44fd34c92f78d9904e/31ca8/avatar.jpg 3x"}}},"headerImage":{"relativePath":"pages/testing-isolated-fragments-with-koin/headerImage.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAECAYAAACOXx+WAAAACXBIWXMAAAsSAAALEgHS3X78AAAAvklEQVQY02NggIIvV/LB9Pvb7WzfrmRs+Hw+vQgmt35mb+DqSS2fV01ujwLxF3dWMZ0/eYwBL/hwNIURRL+7USf8aVfQn/frfLp0p/xnD3JSZ7/14LHdvYePXt66ez8dpObk8WOMj1++wW+gMsiVW3zB7OcLfNVfLfbSALFvCTIwTMwINFrSWRU2rSzFqSXahQkkPqU4kYEo8Hm9F5x9Zl4qY//kOQxT8iI1OqKd5SZmhcouaCoEy03OjcCqHwADLUswHlaKbgAAAABJRU5ErkJggg==","width":1280,"height":280,"src":"/static/a77f9e0124a66d21b204dc560d751ee8/26421/headerImage.png","srcSet":"/static/a77f9e0124a66d21b204dc560d751ee8/26421/headerImage.png 1x"}}}},"html":"

UI-tests are a necessary part of our development routine in AUTO1. Although that wasn’t the case a while ago. One of the issues, which was blocking developers from writing and running UI-tests, was the need to set up UI-tests. Of course, to test the screen in isolation you need to have some architecture, employ the SOLID principles, especially Dependency inversion. With proper architecture it’s easy to substitute dependencies inside screen under the test with fakes or mocks. But even with good architecture you need to provide these mock implementations. And it’s not always obvious how to make your DI framework to provide mocks for tests.

\n

Recently we started a brand new project from scratch. And we decided to give Koin a try. Why Koin and not Dagger? — Just because Koin offers concise and minimalistic syntax and it uses Kotlin. Also, important for us was the question of code generation. Although Google announced that in new versions of Android Gradle plug-in they made kapt incremental, we still didn’t want to add this extra burden to our build process. Simultaneously Koin 2.0 was announced at the same time. That’s how we found ourselves facing question: how to provide mock dependencies to our UI-tests?

\n

Let’s start with fragment…

\n

Inside our fragments we inject number of dependencies, typically they include viewmodels:

\n
private val fragmentViewModel: EmailLoginFragmentViewModel by viewModel()
\n

That’s exactly what we want to substitute with mocks. To do so we need a custom Application class, which will start Koin with our test module, not the real one.

\n
class KoinTestApp: Application() {\n  override fun onCreate() {\n    super.onCreate()\n    startKoin {\n    androidLogger()\n    androidContext(this@KoinTestApp)\n    modules(emptyList())\n    }\n  }\n\n  internal fun injectModule(module: Module) {\n    loadKoinModules(module)\n  }\n}
\n

We start our Koin instance inside onCreate() method, and we pass emptyList() of modules. We also have a method injectModule() which accepts our test module. Since we already started our application, we are using method loadKoinModules() to add this test module to the graph.

\n

How to run custom application

\n

But we need to somehow start our test application class. We cannot use AndroidManifest inside androidTest folder, because manifest merger will ignore it if we set our test application inside Manifest like this: <application name=\".KoinTestApp\">. So we could use a custom test runner for that purpose. AndroidJUnitRunner class has a newApplication() method. And we can pass our test application name there, making our test runner to start our test application instead of a real one.

\n
class KoinTestRunner: AndroidJUnitRunner() {\n  override fun newApplication(\n    cl: ClassLoader?, \n    className: String?, \n    context: Context?): Application {\n      return super.newApplication(cl, KoinTestApp::class.java.name, context)\n  }\n}
\n

Then we need to change our build.gradle inside our app module:

\n
android {\n  defaultConfig {\n    …\n    …\n    testInstrumentationRunner “path.to.your.package.KoinTestRunner”\n  }\n}
\n

And don’t forget to disable animations for UI-tests:

\n
android {\n  testOptions {\n    animationsDisabled = true\n  }\n}
\n

Test rule for fragments

\n

We are almost done. We only need to write our test and provide our test module to it. We are using our custom fragment test rule, which provides a couple of methods for submitting your fragment and mock test module. But you could use the same approach for activities.

\n
abstract class FragmentTestRule<F: Fragment>: \n  ActivityTestRule<FragmentActivity>(FragmentActivity::class.java, true) {\n  \n  override fun afterActivityLaunched() {\n    super.afterActivityLaunched()\n    activity.runOnUiThread {\n      val fm = activity.supportFragmentManager\n      val transaction = fm.beginTransaction()\n      transaction.replace(android.R.id.content, createFragment())\n        .commit()\n    }\n  }\n  \n  override fun beforeActivityLaunched() {\n    super.beforeActivityLaunched()\n    val application = InstrumentationRegistry.getInstrumentation()\n      .targetContext.applicationContext as KoinTestApp\n    application.injectModule(getModule())\n  }\n\n  protected abstract fun createFragment(): F\n  \n  protected abstract fun getModule(): Module\n  \n  fun launch() {\n    launchActivity(Intent())\n  }\n}\n\nfun <F: Fragment> createRule(fragment: F, module: Module): FragmentTestRule<F> = \n  object: FragmentTestRule<F>() {\n    override fun createFragment(): F = fragment\n\n    override fun getModule(): Module = module\n  }
\n

I am using default Android FragmentActivity to host fragment under test. So we need to mention this activity in the manifest. For that purpose I created instance of AndroidManifest inside debug variant in my app module:

\n
<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<manifest\n  xmlns:android=\"http://schemas.android.com/apk/res/android\"\n  package=\"com.github.olegosipenko.kointestsample\">\n  <application>\n    <activity\n      android:theme=\"@style/AppTheme\"\n      android:name=\"androidx.fragment.app.FragmentActivity\"/>\n  </application>\n</manifest>
\n

Test itself

\n

Finally, our test class could look like this:

\n
@RunWith(AndroidJUnit4ClassRunner::class)\nclass EmailLoginFragmentTest {\n  private val fragmentViewModel: EmailLoginFragmentViewModel = mockk(relaxed = true)\n  private val fragment = EmailLoginFragment()\n\n  @get:Rule\n  val fragmentRule = createRule(fragment, module {\n    single(override = true) {\n      fragmentViewModel\n    }\n  })\n\n  @Test\n  fun testBasicInvocation() {\n    onScreen<EmailLoginForm> {\n      emailField { typeText(EMAIL) }\n      passwordField { typeText(PASSWORD) }\n      loginButton.click()\n      verify {\n        fragmentViewModel.loginWithCredentials(EMAIL, PASSWORD)\n      }\n    }\n  }\n\n  class EmailLoginForm: Screen<EmailLoginForm>() {\n    val emailField = KEditText { withId(R.id.textFieldEmail) }\n    val passwordField = KEditText { withId(R.id.textFieldPassword) }\n    val loginButton = KButton { withId(R.id.buttonLogin) }\n  }\n}\n\nprivate const val EMAIL = \"some@email.com\"\nprivate const val PASSWORD = \"password\"
\n

Here we are using Kakao, it’s a nice tool to simplify working with Espresso tests. It provides implementation of Page Object pattern and allows you to abstract interactions with your UI behind the abstract Screen objects. Using Kakao you can improve maintainability of your tests, so if you are not using it yet, give it a try. Also we are using Mockk for mocking, which uses all the power Kotlin gives us.

\n

That’s how with Koin, Kakao and Mockk you could easily test your fragments in isolation, making your life easier.

\n

You can check the source code for the sample here.

","fields":{"slug":"/testing-isolated-fragments-with-koin/","tags":["android","koin","kakao","mockk","espresso","android fragments"]}}},{"node":{"id":"800a446b-0861-54ed-a2e3-a36e545abef1","frontmatter":{"category":"Coding","title":"The concept of SubForms - with React and Formik","date":"2019-07-02","summary":"Create forms at scale with Formik and Yup","thumbnail":{"relativePath":"pages/the-concept-of-subforms-with-formik/thumbnail.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAQCAYAAAAWGF8bAAAACXBIWXMAAAsSAAALEgHS3X78AAABcklEQVQ4y51UzcqCQBSdN+st3AluXOTel6htL2FBuDDaBFGRZGSEuOiHoKI2RbXsR/N83AFjkMbsuzCMjHfOnHvOnWHIRJIk71k2xLxssE+LsmSK1+uVC8pkYLTxdrvh8XjwOf3+dqgU8H6/o9PpYDAYwPd9TCYTuK6L6XSaCyoFfD6fsCwL3W4Xi8UC8/kc2+2Wg8dx/DsgMWw0GqjX6+j3+2i1WhiPx/8vmRjato3hcIjdbof9fs+Zinm5DLPOkQnNZpOzdBwHYRi+zcpzmmXZiQnX6xXL5RKbzYbrl2qXzRX3sE+aHA4HXC6XjxpFUYTj8SjVkokNS3E6naCqKi8zCAKsViveKqTlaDTi7VOpVNBut7nOUoYp4Hq9hmEYqNVqKJfLUBQF1WqVH6LrOkzThKZpKJVKOJ/P30umUkk3SiZDqFVIgtlsxtc9z0Ov1+P9mZYu6smK3mFZSDUUHUvbglxN2yQ76F8hl/OerSJP2R9ITMR6qwehxQAAAABJRU5ErkJggg==","width":395,"height":325,"src":"/static/631a33919834353b3dda8e8578592a5b/b3029/thumbnail.png","srcSet":"/static/631a33919834353b3dda8e8578592a5b/b3029/thumbnail.png 1x,\n/static/631a33919834353b3dda8e8578592a5b/8d141/thumbnail.png 1.5x,\n/static/631a33919834353b3dda8e8578592a5b/ee72c/thumbnail.png 2x"}}},"authorName":"Nicholas Peretti","authorDescription":"Software Engineer","authorAvatar":{"relativePath":"pages/the-concept-of-subforms-with-formik/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAMFBP/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAcKk+4zgAAf/xAAZEAEAAgMAAAAAAAAAAAAAAAACAQMQEiD/2gAIAQEAAQUCJlJVI4M6oXGs8f/EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQMBAT8BH//EABQRAQAAAAAAAAAAAAAAAAAAACD/2gAIAQIBAT8BH//EABsQAQACAgMAAAAAAAAAAAAAAAEAAhAgITFC/9oACAEBAAY/AgO2cmBlvTbX/8QAGxABAQACAwEAAAAAAAAAAAAAAREAIRAgQVH/2gAIAQEAAT8hBOpDNUohZx7hMll+40df/9oADAMBAAIAAwAAABBDwAD/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAcEAEBAAICAwAAAAAAAAAAAAABESFBADEQIFH/2gAIAQEAAT8QxBALK8VVkQLA7X54WBCyOzfKvBj0MduVz6//2Q==","width":50,"height":50,"src":"/static/9347eae31d09e05f032d02892f955d38/d2d31/avatar.jpg","srcSet":"/static/9347eae31d09e05f032d02892f955d38/d2d31/avatar.jpg 1x,\n/static/9347eae31d09e05f032d02892f955d38/0b804/avatar.jpg 1.5x,\n/static/9347eae31d09e05f032d02892f955d38/753c3/avatar.jpg 2x,\n/static/9347eae31d09e05f032d02892f955d38/31ca8/avatar.jpg 3x"}}},"headerImage":{"relativePath":"pages/the-concept-of-subforms-with-formik/headerImage.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAIAAABM9SnKAAAACXBIWXMAAAsSAAALEgHS3X78AAAAUUlEQVQY06WOIQ4AMQgE+/8nYSoQaExFRQmSH4AovX7gkiM3djOZbecH7W2IiDGGqu69y7K7997NrFzOzLXWnFNEbrwm36vMjIgAQETl2194AJcvYFpZxM6mAAAAAElFTkSuQmCC","width":1280,"height":384,"src":"/static/8ce613d93c9ee2f1e3b248f2aabe8efc/26421/headerImage.png","srcSet":"/static/8ce613d93c9ee2f1e3b248f2aabe8efc/26421/headerImage.png 1x,\n/static/8ce613d93c9ee2f1e3b248f2aabe8efc/34a7a/headerImage.png 1.5x"}}}},"html":"

In the last few months I’ve been working a lot with forms. I had to do a large refactoring in one of AUTO1’s applications, and through all the research, I also encountered the term subform. The concept was troublesome not only for me.\nI found more than one issue on the Formik repository about developers asking for help.\nIn this article I'll try to clarify this concept and show you how to use it properly.

\n

What the heck is a subform anyway?

\n

If you’ve ever worked on a medium-large project with a reasonable amount of forms, you might have probably noticed that you could potentially reuse a lot of fields, sometimes even entire forms!\nThat’s the core of the subform concept: reusable components (fields or group of fields) that can be composed together to create bigger forms.

\n

At the beginning I had a lot of questions in my mind like:

\n\n

These questions and many others appeared while I was refactoring existing codebase. Many similar questions created by other developers appeared in the Formik repository as issues.

\n

Don’t get me wrong: implementing scalable validation for a single form is not that hard. The complicated thing is to keep validation and state flexible while you write your subforms. What does it mean? We’ll see that in a moment.

\n

How Formik can help

\n

Formik is an amazing library, one of the bests for this job, and here’s what it brings to the table:

\n

Auto-connected Field component

\n

Formik provides a Field component that, thanks to the React Context API, will be automatically connected to the Form component that wraps our Field, regardless of how deep our Field is in the tree.

\n

Yup validation schema

\n

Yup is a library to create validation schemas. This library is very similar to the prop-types of React, so it is very simple to start using it.\nFormik supports Yup natively, so you just need to define the schema and pass it to Formik, it will take care of the rest.

\n

So this is how Formik will make our life easier not only while creating subforms but also when we’ll need to maintain them!

\n

Enough talking, show me the code

\n

Let’s create our first subform! Before we start, we need to clarify what our subform will be responsible for:

\n\n

In our case, we're going to create a form that allows to create a new user.\nWe'll need to display the following fields:

\n\n

Just keep it simple. So, let’s create our NewUserSubForm directory and our fieldsNames file. It’s just a file that exports constants, don’t worry. It will look like this:

\n
//  UserSubform/fieldsNames.js\nexport const FIRST_NAME = 'firstName'\nexport const LAST_NAME = 'lastName'\nexport const EMAIL = 'email'\nexport const PASSWORD = 'password'
\n

Defining the validation schema

\n
//  UserSubform/validationSchema.js\nimport * as yup from 'yup'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nconst MIN_PASSWORD_LENGTH = 8\n\nconst REQUIRED_MESSAGE = 'This field is required'\nconst INVALID_EMAIL_FORMAT = 'Invalid email format'\nconst PASSWORD_TOO_SHOWRT = `The password must be at least ${MIN_PASSWORD_LENGTH} characters long`\n\nexport default yup.object({\n  [FIRST_NAME]: yup.string().required(REQUIRED_MESSAGE),\n  [LAST_NAME]: yup.string().required(REQUIRED_MESSAGE),\n  [EMAIL]: yup\n    .string()\n    .required(REQUIRED_MESSAGE)\n    .email(INVALID_EMAIL_FORMAT),\n  [PASSWORD]: yup\n    .string()\n    .required(REQUIRED_MESSAGE)\n    .min(MIN_PASSWORD_LENGTH, PASSWORD_TOO_SHOWRT),\n})
\n

We’ve just defined our validation schema regardless the markup of our form. This file just contains all the validation logic and has one responsibility. If in the future we’ll need to add some more validation options, we’ll just need to change it here.

\n

Now it’s time for the default values. Initial values are required from Formik because it uses controlled inputs. So, if you don’t specify those values, you’ll get and error from React as soon as you try to change the content of the input.

\n
//  UserSubform/defaultValues.js\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nexport default {\n  [FIRST_NAME]: '',\n  [LAST_NAME]: '',\n  [EMAIL]: '',\n  [PASSWORD]: '',\n}
\n

Rendering the subform

\n

And now the master piece: The React component. Remember: we just need to use the Fields and not the Formik or the Form components.

\n
//  UserSubform/index.js\nimport React, { Fragment } from 'react'\nimport { Field, ErrorMessage } from 'formik'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nexport default class NewUserSubForm extends React.Component {\n  render() {\n    return (\n      <Fragment>\n        <Field component=\"input\" name={FIRST_NAME} />\n        <ErrorMessage name={FIRST_NAME} />\n        <Field component=\"input\" name={LAST_NAME} />\n        <ErrorMessage name={LAST_NAME} />\n        <Field component=\"input\" name={EMAIL} />\n        <ErrorMessage name={EMAIL} />\n        <Field component=\"input\" name={PASSWORD} />\n        <ErrorMessage name={PASSWORD} />\n      </Fragment>\n    )\n  }\n}
\n

And that’s it. In this phase we can test every single part of our subform: validation, the default values schema and the React component.

\n

A piece of advice: Formik sets the fields’ values in its state using the name property, but the cool thing is that it uses it like the Lodash set function. It means that we can write the name of a field like this: user.firstName. In this way Formik will create an object in its state called user, and then a property inside of user called firstName that will contain the value of our field.

\n

This mechanism gives us power to improve the flexibility of our subform. How?

\n

Making a subform flexible

\n

Let’s edit our component in a way that it will accept an optional property called namespace. If received, the component will prepend the namespace to every field name. In this way it will be easier to wrap all the subform’s values under a certain object in the main form.

\n
//  UserSubform/index.js\nimport React, { Fragment } from 'react'\nimport PropTypes from 'prop-types'\nimport { Field, ErrorMessage } from 'formik'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nexport default class NewUserSubForm extends React.Component {\n  static propTypes = {\n    namespace: PropTypes.string,\n  }\n\n  withNamespace(fieldName) {\n    const { namespace } = this.props\n\n    return namespace ? `${namespace}.${fieldName}` : fieldName\n  }\n\n  render() {\n    const { withNamespace } = this\n\n    return (\n      <Fragment>\n        <Field component=\"input\" name={withNamespace(FIRST_NAME)} />\n        <ErrorMessage name={withNamespace(FIRST_NAME)} />\n        <Field component=\"input\" name={withNamespace(LAST_NAME)} />\n        <ErrorMessage name={withNamespace(FIRST_NAME)} />\n        <Field component=\"input\" name={withNamespace(EMAIL)} />\n        <ErrorMessage name={withNamespace(FIRST_NAME)} />\n        <Field component=\"input\" name={withNamespace(PASSWORD)} />\n        <ErrorMessage name={withNamespace(FIRST_NAME)} />\n      </Fragment>\n    )\n  }\n}
\n

We don’t need to do that with the other parts of the subform, the main form will be responsible for that. And, about the main form, let’s see how to implement it!

\n

The main form

\n

Finally, we’re going to create our main form component. Let’s define its responsibilities just like we did with our subform. The main form will be responsible for:

\n\n

It's a lot of responsibilities, and that’s all right. The main form represents a specific point in the UI/UX where the user needs to insert some data. In our case, it could be a registration form, but it could also be a registration combined with a purchase, just like when you buy something from amazon and you agree to sign up in the process.

\n

The point is: A Form is a unique component that represent a specific use case, so it has to be designed accordingly. That’s why it makes no sense to create a “god-form” component with hundreds of props that decides which endpoint the form is going to use. It just creates useless complexity.

\n

In my opinion, the best way to organize this approach is to create a folder where you’ll store all your subforms. Every subform will be represented by its directory and it will contain all its parts: validation, values, fields and the React component.\nA main form, instead, should be created ad-hoc to fit the needs of a certain use case, for example inside a certain route.

\n

So, with that in mind, let’s proceed to the implementation. We’ll have our directory called registrationForm and it will have the same parts of a subform:

\n

Fields Names

\n
//  CreateNewUserRoute/form/fieldsNames.js\nexport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './subforms/NewUserSubForm/fieldsNames'\nexport const USER = 'user'
\n

Validation

\n
//  CreateNewUserRoute/form/validationSchema.js\nimport * as yup from 'yup'\nimport { USER } from './fieldsNames'\nimport userValidationSchema from './subforms/NewUserSubForm/validationSchema'\n\nexport default yup.object({\n  [USER]: userValidationSchema,\n})
\n

Default values

\n
//  CreateNewUserRoute/form/defaultValues.js\nimport { USER } from './field Names'\nimport userDefaultValues from './subforms/NewUserSubForm/defaultValues'\n\nexport default {\n  [USER]: userDefaultValues,\n}
\n

The React component

\n
//  CreateNewUserRoute/form/index.js\nimport React from 'react'\nimport { Formik, Form } from 'formik'\nimport NewUserSubForm from './subforms/NewUserSubForm'\nimport validationSchema from './validationSchema'\nimport defaultValues from './defaultValues'\nimport { USER } from './fieldsNames'\nimport ErrorBanner from 'path/to/components/ErrorBanner'\n\nexport default class NewUserSubForm extends React.Component {\n  state = {\n    unknownErrors: null,\n  }\n\n  onSubmit = async (values, { setSubmitting, setErrors }) => {\n    try {\n      //  Send values somehow\n      await sendForm(values)\n    } catch (e) {\n      //  Map and show the errors in your form\n      const [formErrors, unknownErrors] = mapErrorsFromRequest(e)\n\n      setErrors(formErrors)\n      this.setState({\n        unknownErrors,\n      })\n    } finally {\n      setSubmitting(false)\n    }\n  }\n\n  render() {\n    const { unknownErrors } = this.state\n\n    return (\n      <Formik onSubmit={this.onSubmit} initialValues={defaultValues} validationSchema={validationSchema}>\n        {() => (\n          <Form>\n            {unknownErrors && <ErrorBanner errors={unknownErrors} />}\n            <NewUserSubForm namespace={USER} />\n          </Form>\n        )}\n      </Formik>\n    )\n  }\n}
\n

And that’s it! Of course, this is a very simple example, you could have different needs.

\n

Helpful advices on creating subforms

\n

I want to leave you with some advices that helped me while I was refactoring my codebase. It's good to have them in mind to ease the process of code refactoring.

\n

A subform should have only first level values in its state

\n

A subform should have only first level values in its state, which means that, when you design a subform, you shouldn’t get crazy about the shape of its values. It should be a flat object and every key should contain the field value.\nThis way it’s so much easier to write validations, default values and error handling (and why not, also the React component).

\n

You can avoid this advice only when you are using a subform into you subform. For example, let’s say you have an address subform. It has a lot of fields and a complex validation schema. In that scenario all the logic will be handled by the address subform and you’ll just need to orchestrate it in your own subform, just like you would do in the main form.

\n

Keep the validation schema extensible and scalable

\n

I didn’t do it in this article but the idea is to export a function instead of a schema. This function will accept parameters that will define the schema that you’ll get.\nIn this case you can toggle the “required” validation in some cases, or other kinds of validation.

\n

Example: let’s say that we want to make the “lastName” field optional, but not always. That’s how you could define your schema:

\n
//  UserSubform/validationSchema.js\nimport * as yup from 'yup'\nimport { FIRST_NAME, LAST_NAME, EMAIL, PASSWORD } from './fieldsNames'\n\nconst MIN_PASSWORD_LENGTH = 8\n\nconst REQUIRED_MESSAGE = 'This field is required'\nconst INVALID_EMAIL_FORMAT = 'Invalid email format'\nconst PASSWORD_TOO_SHOWRT = `The password must be long at least ${MIN_PASSWORD_LENGTH} characters`\n\nexport default function validationSchema(mandatoryFields = { [LAST_NAME]: true }) {\n  return yup.object({\n    [FIRST_NAME]: yup.string().required(REQUIRED_MESSAGE),\n    [LAST_NAME]: yup.lazy(() => (mandatoryFields.lastName ? yup.string().required(REQUIRED_MESSAGE) : yup.string())),\n    [EMAIL]: yup\n      .string()\n      .required(REQUIRED_MESSAGE)\n      .email(INVALID_EMAIL_FORMAT),\n    [PASSWORD]: yup\n      .string()\n      .required(REQUIRED_MESSAGE)\n      .min(MIN_PASSWORD_LENGTH, PASSWORD_TOO_SHOWRT),\n  })\n}
\n

Now you have a scalable validation schema.\nIn this way you can always decide if a certain field is required or not. Also, you can extend that function to add parameters. In this way, if your subform scales, you’ll just need to add parameters and adjust the code in the subform accordingly, but every single form that uses your subform won’t be affected by these changes because everything is retro compatible.

\n

Conclusions

\n

Dealing with forms is not easy. Of course, there are simple cases, but there are also trickier ones. This was my way to organize the code, and of course, it’s not perfect.\nI’m sure there are other thousands amazing ways to solve this problem better than this, but for now, this is the best way I’ve found to keep everything testable, maintainable and scalable.

\n

I hope this will help you, and if you have a better approach, I’m looking forward to reading it!\nUntil next time, happy hacking!

","fields":{"slug":"/the-concept-of-subforms-with-formik/","tags":["formik","forms","react","subforms"]}}},{"node":{"id":"61712d64-9fb5-5c05-8e36-ae37d9bf956d","frontmatter":{"category":"Coding","title":"Take a deep Bref, and run your Symfony app serverless","date":"2019-06-24","summary":"Yet another case study of developing serverless apps with PHP.","thumbnail":{"relativePath":"pages/php-serverless-with-bref/blog.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAABAACA//EABUBAQEAAAAAAAAAAAAAAAAAAAAC/9oADAMBAAIQAxAAAAFvIGahEOP/xAAYEAADAQEAAAAAAAAAAAAAAAAAARMCEv/aAAgBAQABBQLlG9clCrRTXVWf/8QAFREBAQAAAAAAAAAAAAAAAAAAEBH/2gAIAQMBAT8Bh//EABURAQEAAAAAAAAAAAAAAAAAAAAR/9oACAECAQE/Aar/xAAXEAADAQAAAAAAAAAAAAAAAAAAESAh/9oACAEBAAY/AsFP/8QAGRABAQEBAQEAAAAAAAAAAAAAAREAMSFx/9oACAEBAAE/IROt9TIugIGZBbfeZrv/2gAMAwEAAgADAAAAEC//AP/EABcRAAMBAAAAAAAAAAAAAAAAAAABESH/2gAIAQMBAT8Q0qiI/8QAFhEBAQEAAAAAAAAAAAAAAAAAAREQ/9oACAECAQE/ECGLn//EAB4QAQEAAQMFAAAAAAAAAAAAAAERACExUUFhgcHR/9oACAEBAAE/EG97q6XxipRA8SmUKC93GIysvGBJkuwRxavr5n//2Q==","width":488,"height":325,"src":"/static/af9a71b099ce7da56586de41ed112b03/a2998/blog.jpeg","srcSet":"/static/af9a71b099ce7da56586de41ed112b03/a2998/blog.jpeg 1x"}}},"authorName":"Wojciech Oroński","authorDescription":"Wojciech is a Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/php-serverless-with-bref/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACtklEQVQ4y4VUvWtTURS/SV7sUIeK+g/o6GSS9/Ly8pKmKc0g2gqBVkrBQAcpOggOuhVxcbAG/FpcOgqOguAg1G6COIifCKKIijo5qDWf/n4v57QHFx/8uOfde+7vnm9Xq8WZqalKmmupVNzl5Mvn81WgncvlXmHtAh3IH7CuA7NhONKNojBbrUbbHI7C5GSc4YEQFYGNQqEw9H1/CHkohD3K3JP950CTd0gUxyWvXq+kkx9DtqyXhGBLiQnKtJQwD17l3Xq9mq5UooxTN0lmLOooMdx8B/k+cA94RiJD3AuCgPJ1cpTLJc8ZN617JPqD9UwQ+OMaV3jiYe8I8ElI6UVXSJedScCGWJe8KvIpns3M1NLlcpgl4JIn+hEe7IsBPZLj/zPWvUk25bW+gIdffL+wZ3FxLsX4Ms4EY2RCtGli3RErz/KgLQd0tS/kL+Hq7lar6UiihAQtFcI75p569dDBmjdilbXwG4j38WIcRx5Ki3WWYXkZtx+phdAfyL0fTpMADMxLt0dkJY8kUvyZf0rstAnVQEmdEClhVwgv81IYBmMsVnWXTYAOGRPCEyaGO4QQPoq5AyWH/BPrIW0tbSu6u7R0PAWicei81tIxLv/mS+smuENTV3d33C4n1pkMnxedri0drI95OGuDK6+pQkutNNnNQ+eX8Wpoyuai49SA8MJU/kALXPZCEs3PH03hfz9Ivsr+FhNi3GWYDmqnNK3bfK1YLPJ/k/W4sHAsZbJ7g2fUUWKx7pL2pyquCcl74CZwDt0ywTPGDshqt8hUagNP5c6DRqOWZps6DsdkMI5Ib8lQOOn+87Fl2aKcA5qsZNowe8wkV1FcYfqB75CvAXP50XcYaDDwOHsrybiyurqy7SlnYjJgWbx0RTOJ2ExA+QLwRDOqWZVWXQMOUHd6upqmZSQj119HCq8aFJGDXQAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/d606e0fe3a0b15d5494a19629b2b3022/45876/avatar.png","srcSet":"/static/d606e0fe3a0b15d5494a19629b2b3022/45876/avatar.png 1x"}}},"headerImage":null},"html":"

tl;dr

\n

Since AWS lambda supports custom runtimes you can use PHP to create AWS lambda functions.\nBut it comes with two major drawbacks

\n\n

With bref.sh these problems will disappear. Let's try to run Symfony framework as a serverless app.

\n

What is serverless ?

\n

What does it mean that something is Serverless? Does it mean there are no servers at all?\nOf course not. There are still servers behind scenes, but you don't have to worry about maintaining the servers and reserve their capacity. Your infrastructure is scaled automatically, and you just pay only for what you use.\nDoes this kind of architecture have only pros or there are some drawbacks?\nLet's leave the discussion of all the smaller pros and cons of serverless applications for another article.\nI will focus on the biggest drawback which is an impact on our code. Frameworks that we use in our daily PHP development are not ready to work with lambdas because they were not designed for those kinds of architecture.\nImagine you are a PHP developer and you want to run serverless microservice using one of the most popular frameworks like Symfony or Laravel. Imagine you can develop your microservice locally and just deploy it as lambda functions. Is it possible with PHP\nprogramming language? Yes, it is. Even if AWS Lambda does not have native support for PHP.

\n

Here comes Bref.

\n

Since AWS lambda supports custom runtimes, you can create one for PHP and run your lambda functions written in that language. There are plenty of articles which describe the process of creating custom runtimes, but believe me it's time-consuming and not super easy.\nLet's take a look at the bref.sh project and try to run a simple Symfony app as lambda.

\n

I assume you have knowledge about Symfony framework, or you have an existing project and some knowledge about AWS platform.

\n

Before we start please make sure you have installed:

\n\n

First, we have to add Bref to our project with composer require mnapoli/bref\nThen from your project root command line run vendor/bin/bref init\nIt will create two additional files in your project template.yaml and index.php.\nYou can remove index.php for this moment, as it will be unnecessary.

\n

As you may know, AWS lambda filesystem is read-only excluding /tmp dir\nSo we have to extend src/Kernel.php of Symfony with these two functions

\n
    public function getLogDir(): string\n    {\n        // When on the lambda only /tmp is writeable\n        if (getenv('LAMBDA_TASK_ROOT') !== false) {\n            return '/tmp/log/';\n        }\n\n        return $this->getProjectDir().'/var/log';\n    }\n\n    public function getCacheDir(): string\n    {\n        // When on the lambda only /tmp is writeable\n        if (getenv('LAMBDA_TASK_ROOT') !== false) {\n            return '/tmp/cache/'.$this->environment;\n        }\n\n        return $this->getProjectDir().'/var/cache/'.$this->environment;\n    }
\n

Let's take a closer look at template.yaml

\n
Resources:\n    Website:\n        Type: AWS::Serverless::Function\n        Properties:\n            FunctionName: 'symfony-website'\n            CodeUri: .\n            Handler: public/index.php\n            Timeout: 30 # in seconds (API Gateway has a timeout of 30 seconds)\n            MemorySize: 1024\n            Runtime: provided\n            Layers:\n                - 'arn:aws:lambda:us-east-1:209497400698:layer:php-73-fpm:1'\n            Events:\n                HttpRoot:\n                    Type: Api\n                    Properties:\n                        Path: /\n                        Method: ANY\n                HttpSubPaths:\n                    Type: Api\n                    Properties:\n                        Path: /{proxy+}\n                        Method: ANY\n\n    Console:\n        Type: AWS::Serverless::Function\n        Properties:\n            FunctionName: 'symfony-console'\n            CodeUri: .\n            Handler: bin/console\n            Timeout: 120 # in seconds\n            Runtime: provided\n            Layers:\n                - 'arn:aws:lambda:us-east-1:209497400698:layer:php-73:1' # PHP\n                - 'arn:aws:lambda:us-east-1:209497400698:layer:console:1' # The \"console\" layer
\n

Thanks to the configuration stored in template.yaml, Bref will create CloudFormation with two most important resources which are lambda functions that will handle all the requests. One called symfony-website and the second symfony-console, both names are defined under FunctionName key.\nTheir execution will be available thanks to API Gateway which also will be created during the deploy process.\nFor testing purpose, you don't have to change template file much. Please remember to check your AWS default region if it's different than us-east-1 just change it in all Layers key of your Resources.

\n

OK. Let's check how it works and deploy.\nFirst, we need to create s3 bucket per our app with simple AWS CLI command aws s3 mb s3://<bucket-name>\nThen upload the code and generate the stack configuration:

\n
sam package \\\n    --output-template-file .stack.yaml \\\n    --s3-bucket <bucket-name>
\n

Finnaly deploy our microservice:

\n
sam deploy \\\n    --template-file .stack.yaml \\\n    --capabilities CAPABILITY_IAM \\\n    --stack-name <stack-name>
\n

After we receive output \"Successfully created/updated stack - \" our serverless microservice is ready to use.\nLet's check our resources in the AWS console. SAM deploy created the corresponding AWS Lambda Functions and API Gateway resources.

\n

Lambda Functions:

\n

\n \n \n \n \n

\n

API Gateway:

\n

\n \n \n \n \n

\n

Just check your execution url in API Gateway:

\n

\n \n \n\n

Other drawbacks:

\n\n

But still I think this is a great starting point and I believe it can solve two major problems of the \"serverless PHP\":

\n\n

Further info

\n","fields":{"slug":"/php-serverless-with-bref/","tags":["Symfony","PHP","serverless"]}}},{"node":{"id":"4847d312-706f-5678-9e9e-1a333f42095e","frontmatter":{"category":"Coding","title":"How we use ES6 generators to simplify our React Redux application code","date":"2019-04-04","summary":"How we use ES6 generators instead of thunk to simplify our React Redux application code and improve productivity","thumbnail":{"relativePath":"pages/es6-generators-react-redux/thumbnail.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAnLAAAJywEjDYgKAAADmUlEQVQ4y3VV2W/MURQemUY7+7Szd2Y681umLSHWB4TW1nqo5YUisYVIVGsJEXtrryUIam9Va0nQ1hoULYrEFhL8ASS2+A94+Zxzfp0xpR5ucnPPud893znfOdekqioUWqqiyFI0DVooAN2bDTUagaLrYpdFez5jm5YbFF+5k8SgZUoHVFQCi4QRmlIOb8V65A0fAd2Xk7rA++iIMfAsq0Zu2XRo4dweoClAOaDXtYAPgTlLkNn6Cq5tx5HV+hLu6sMUVRRqLM84a3kB95Y6ZLa9RnDGAmhBP4HqBsP0CIUq0bCdvInQtNnIt5gRGzgIjgMXYD/SAvvRNjj3NSPefwAKMvsgMHuxnHMQDJiKUPmLsvVMOyKjxxtUiL4WCsL89BsyHn+WaNS8qPiHJ5bJ42o89ifHEmEyqXpCAOxHWxEZW4qEwyIAjsOXkb1uL7I37Ifj4EWJKGHPQrhkChyHLlEeQ1AS+amgTCm65Kj7veJkO3UL2RsPwNr0AKYPv2Bt7oDlXKexb7wn4Lb623DsPw/d4+7Oo9YdYTwu1fItXi0AfW++Q+a1t/Cs2i5Uo6PHIV7QD0pBISLFpTA/+w7Pyq3ix8t69j7886sk/xIh64oTb23qoAsliA0eRrm5IZF6K9ZRdWNw7m2Cc/cZ0SBLhqnbTtxAbNBQhCeUwXKxS+wsOZNv0SoyXpfQOeGsNeeeRmQ8/ER7j9Ay/YAsZ22DpMX85AtcO06SwHMMoZMfp8A/r5IAF66U1wSQohFAei2j86OxJ6kkAV27TgtgRtdn2teLXYLwM+Ad+OcuTaNMuYiOKibtDRY5sMa8S9bKBb7MEWmRXIMy+QvlIcNJEZNgucCUG6QWqaLwZcv5R1SQN9IF3sqNMHd9kUdEFrQiYyZQob7CW7VJ/Pre+iCF9C9Y/qcohiAN2bCI7ceukrjvwl1TJw+Y3v+EpblTpJPcuzcfEmkxC06BFvSlySZN2Fwle12rVDvhssnUsR+5IsLOWVNLthY5SzitiBRNlHY0JlLiL2EnW49esTa2i/ak9ciZRcsVZ6rcFQJANga0NtxNa9tk6/UYDiGiQsNh6izkZ5kRL+wH187TUiSWFlc/Xtgf+TQceNJwetK7pNfx5Z9TIeOLx1bWledw1dYbo4kk5dp5Ss7YxoULzFxk5E/vZXz1GLAUIcsjOrLI0Fo3JRmwVHXPihqEJpf/f8D+9wsg8H++ADqTgRDq/Qv4DWMLuHUxNoXrAAAAAElFTkSuQmCC","width":325,"height":325,"src":"/static/2bee11a830bacc5ae9006df56b20c33a/b3029/thumbnail.png","srcSet":"/static/2bee11a830bacc5ae9006df56b20c33a/b3029/thumbnail.png 1x,\n/static/2bee11a830bacc5ae9006df56b20c33a/8d141/thumbnail.png 1.5x,\n/static/2bee11a830bacc5ae9006df56b20c33a/ee72c/thumbnail.png 2x,\n/static/2bee11a830bacc5ae9006df56b20c33a/5dfa8/thumbnail.png 3x"}}},"authorName":"Chirag Swadia","authorDescription":"Chirag is a Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/es6-generators-react-redux/avatar.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAUABQDASIAAhEBAxEB/8QAGAABAAMBAAAAAAAAAAAAAAAAAAECAwT/xAAXAQEBAQEAAAAAAAAAAAAAAAAEAwAB/9oADAMBAAIQAxAAAAGIveV5aObk0FjsCs//xAAbEAADAAIDAAAAAAAAAAAAAAAAAQIRIRIiQf/aAAgBAQABBQKnxmHTMGhGz0l9f//EABoRAAICAwAAAAAAAAAAAAAAAAABAhAxMkH/2gAIAQMBAT8B7U9mLB//xAAZEQABBQAAAAAAAAAAAAAAAAAAAQIQMTL/2gAIAQIBAT8BhuRbP//EABsQAAIBBQAAAAAAAAAAAAAAAAARAgEQIDFB/9oACAEBAAY/AmKcEatzCh//xAAbEAEAAwADAQAAAAAAAAAAAAABABEhMVFh8P/aAAgBAQABPyEnrfkRsPknmGxTsy8Dqp82W0JRuRFFn//aAAwDAQACAAMAAAAQ+Dj/AP/EABgRAAMBAQAAAAAAAAAAAAAAAAABETEQ/9oACAEDAQE/EG5ovBkf/8QAGBEAAgMAAAAAAAAAAAAAAAAAADEQESH/2gAIAQIBAT8Q2lCBh//EABwQAQADAQEAAwAAAAAAAAAAAAEAESExUXGB8P/aAAgBAQABPxCyt4AerkU0XY8TkPRXxBSQfql1NpWO/co8qKgOPZRgcYmBU9n/2Q==","width":50,"height":50,"src":"/static/8cf5c37636309895b29dd7ce5f1fcf28/d2d31/avatar.jpg","srcSet":"/static/8cf5c37636309895b29dd7ce5f1fcf28/d2d31/avatar.jpg 1x,\n/static/8cf5c37636309895b29dd7ce5f1fcf28/0b804/avatar.jpg 1.5x,\n/static/8cf5c37636309895b29dd7ce5f1fcf28/753c3/avatar.jpg 2x,\n/static/8cf5c37636309895b29dd7ce5f1fcf28/31ca8/avatar.jpg 3x"}}},"headerImage":{"relativePath":"pages/es6-generators-react-redux/headerImage.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAAsSAAALEgHS3X78AAABWUlEQVQY022QS0/CQBSFOzMtFLCdtsOrPOQRIPJogIILgkY0GqKJG5cu3GqMxIULExOJG/9s/0N7nBbjRiaZnJs7ky/nHEWRx63WRDqVCjRdR344DUV/jPxohlQmC0YpNE2Dqqp/SqkC+8BFuzBHTucRYzSK3xzHOY95SrXXF1KCbK2Fzv1T6L1+ofewQa7egtz/u0Sh6BZO0HBmOLT9SO4iohAYhrED6ooidJMHldUNvJdt6G22Uj/hXtyiO/TgTyeY+j4WiwUKpTxEtomheyVhE4wq64jr5RgKi/86LJZdQQkJ7MEUg8f3cPz2jeHzB+zRHLIKcM5hmqZUE2ktg4bto1NYomZ56BVPo6Yzjwgh8Z8d0G22BZGRWTaH8vIyrK/vUDm7hpoz9kauWqMk8lFphZY4jspGL3HIubUD5oUQjKlB0o+qhokythf21yOhiVLCkg7jOXb4A0fehWKiiDXIAAAAAElFTkSuQmCC","width":757,"height":236,"src":"/static/c49511e796013b6394df522c11a4a9fc/7eaf7/headerImage.png","srcSet":"/static/c49511e796013b6394df522c11a4a9fc/7eaf7/headerImage.png 1x"}}}},"html":"

Background

\n

When I first started working on React/Redux, the very first choice of library I used for handling asynchronous actions was Redux Thunk as its maintained by the authors of Redux, and is very popular as well. Consider the following example, where an action type is being created for incrementing a counter and a corresponding action creator for it.

\n
const INCREMENT_COUNTER = 'INCREMENT_COUNTER';\n\nconst increment = () => ({\n  type: INCREMENT_COUNTER\n})
\n

Using redux thunk, an async action creator can be created as follows

\n
const incrementAsync = () => {\n  return dispatch => {\n    setTimeout(() => {\n      dispatch(increment());\n    }, 1000);\n  };\n}
\n

Pretty simple till now.

\n

The Problem

\n

Redux Thunk is easy to understand, but the problem arises when the application scales, and as the need for dispatching condition-based actions grows with the application, it leads to unmaintainable code. For example, consider the below code

\n
const makeSandwichesForEverybody = () => (dispatch, getState) => {\n  if (!getState().sandwiches.isShopOpen) {\n    return Promise.resolve();\n  }\n\n  return dispatch(\n    makeASandwichWithSecretSauce('My Grandma')\n  ).then(() =>\n    Promise.all([\n      dispatch(makeASandwichWithSecretSauce('Me')),\n      dispatch(makeASandwichWithSecretSauce('My wife'))\n    ])\n  ).then(() =>\n    dispatch(makeASandwichWithSecretSauce('Our kids'))\n  ).then(() =>\n    dispatch(getState().myMoney > 42 ?\n      withdrawMoney(42) :\n      apologize('Me', 'The Sandwich Shop')\n    )\n  );\n};
\n

In the above code, there are too many callbacks and the code does not look clean. Also, it would become complex if more conditions are added to it later on. Then, how to simplify this?

\n

ES6 Generators to the rescue

\n

For our customer-facing websites like wirkaufendeinauto and AutoHero, we are neither using Redux Thunk nor Redux Saga. Rather, we are using a very simple and lightweight npm module Redux Actions Generators developed by our very own Alexander Afonin, who works as a Senior Software Engineer at Auto1.\nIf this module is configured as a middleware in the React application, the above code can be simplified as shown below

\n
const makeSandwichesForEverybody = () => function* ({ getState }) {\n  if (!getState().sandwiches.isShopOpen) {\n    return;\n  }\n\n  yield makeASandwichWithSecretSauce('My Grandma');\n  yield [makeASandwichWithSecretSauce('Me'), makeASandwichWithSecretSauce('My wife')];\n  yield makeASandwichWithSecretSauce('Our kids');\n\n  if (getState().myMoney > 42) {\n    yield withdrawMoney(42):\n  } else {\n    yield apologize('Me', 'The Sandwich Shop');\n  }\n};
\n

Now you are ready to tackle the growing complexity with ease. Also, the code is clean and readable.

\n

One question that might arise is that why not use async/await with thunk actions which could have solved the problem that is being discussed earlier. But, there are advantages of using generators instead of async/await thunks which are listed below -

\n

Testability

\n

Redux Thunk returns promises, which are more difficult to test. Testing thunks often require complex mocking of the fetch API, Axios requests, or other functions. With generators, you do not need to mock functions wrapped with effects. This makes tests clean, readable and easier to write.

\n

Error Handling

\n

The best part about the Redux Actions Generator module is that you can catch all errors at one place, without the need to write multiple try catch blocks in different actions ( you can if you want to, but not mandatory ) With redux-thunk, this is not possible and you have to write error handlers everywhere, which makes the code look less readable.\nTo use this common error catching functionality, you can configure it while creating the store as shown below -

\n
const catchError = error => console.error(error); \nconst middlewares = applyMiddleware(createGeneratorMiddleware(null, catchError));
\n

Once you do this, your actions can become clean and more readable. For example, the below code...

\n
const someAction = () => function* ({ api }) {\n  try {\n    yield api.loadSomeItems();\n  } catch (error) {\n    // handle error logic\n    console.log(error);\n  }\n}
\n

can be replaced with...

\n
const someAction = () => function* ({ api }) {\n  yield api.loadSomeItems();\n}
\n

In this way, all the uncaught errors will be caught with the catchError function.

\n

Enough of Making Sandwiches

\n

Let's get to some real life example on how we use this pattern in our code. Consider the Auto Kaufen page on our wirkaufendeinauto website where we show a list of ads, filters, header, footer and some other content.

\n

For this module, we have some actions like

\n
const loadAds = (filters) =>\n  function*() {\n    // filter is an object with some information like {make: 'Audi', model: 'A3'}\n    \n    const { ads: result } = yield api.findAds(filters);\n\n    // Once we get the ads list, we just save it in the redux store\n  }\n\nconst loadSimilarAds = (filters) =>\n  function*(){\n      //  this is same as loadAds, but the API endpoint from which it fetches data is different\n  }
\n

Similarly, we have other actions like loadDataFromCms, trackGtmEvents etc. Now the action which will be called when the main component mounts, will be as shown below

\n
export const loadPage = () => \n  function*({ getState }){\n      yield loadDataFromCms('header');\n\n      const filters = yield getFiltersFromStore(getState());\n      yield [\n        loadAds(filters),\n        loadMakeModelSubtype(filters),\n        loadDataFromCms('home')\n      ];\n\n      // If no ads found, we try to find similar ads which might interest the user\n      if( getState().ads.length === 0 ) {\n          yield loadSimilarAds(filters);\n      }\n  }
\n

As you can see, in the above code we are making multiple API calls, some are in parallel, and some are fired conditionally based on the results of the previous API calls. This way our redux actions look clean and are easy to debug and maintain.

\n

Summary

\n

So this is how you can make use of the ES6 generators to simplify your React application code. Comments and suggestions are welcome.

","fields":{"slug":"/es6-generators-react-redux/","tags":["auto1","engineering","react","redux"]}}},{"node":{"id":"cd170267-3e38-5f47-8461-d6f6583c5bed","frontmatter":{"category":"Coding","title":"Our love story with Deadlocks (PostgreSQL Edition)","date":"2019-03-15","summary":"How to get rid of deadlocks","thumbnail":{"relativePath":"pages/event-store-deadlock/thumbnail.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAVCAYAAABG1c6oAAAACXBIWXMAAAsSAAALEgHS3X78AAAFFklEQVQ4y5WUeUzTZxjHi+AWnTuYc1mAKaiAIgoI4tR5henqgSBEoOOmCINxKAIiRPFCpK304CyUUikUtIisHC0UhHoy8ZoaY2Lcodv8w3+WxcyMtu937/ubOLNliXuSN2/yvu/zeb/P8b483t8WvjlkjTopTmCMiQzrXBm4uIyu8T94d0ZGdERYf3Ji/JWIUL4hThB1QZiW8dvuvD0kJzv7aSh/Qy89F8p71QKW+Krr1VrUdF3A4RYzxB0jqNYPQ9Oqg+a0AeJTFhxoNqNCN4zx+49x+f4TCKU9yKruQ233JVTVNcBz7hwlB5s21SFd096FhMo+69xY0YR3gsTmFlVuDS9tmzDd+IGs3aO2zYsVWb3ixXbPeIltUXKlLb+uh6h6x8iCBDFxF1RYhXKjtUVvwEezZpbxYmN2XKnoOI/Z0cdsyzOqEJxRRXyTT6DZOE6iDmqJd7yILM+sQlC6nCz7UkEC0+RwjylHUUM/aeq/ShanVOLjqDL7oVYLkhLifuRlZ2f/mlXdD+YYlK7gDmzIb4SyZwzugnJQZfClawwUsFPGzUvp+HD7YdR0X4ZQpCdUDMlXGiEUpjzh5eTkPs/kgBUkiCpYmCjBF0d1OHP+Dg5qzJxTaHEzfJMrqUoF/IRSrN1VD/mZixi99RBKwxiZtf2wvVw3gpgdEdd5aanC7/aqBuAZJ7KzkGhekCruxND1B8iQnsUIdWofvgX/VBnohVSxCKKOUbQN3YS08wIkp0bJbIHI2thtQbD/okZe2OaN3dLTFtCEW4MzFPBmQEknDp00w21HGVbn1EHVdxUbC1TwS5WyiznVm4vUEIr1pETVTz7JVdlqmlpAayzgec1xKVGeNiIwo9YamCb7SyEFitpHOThTxPK5rUQDll9vmtNd1QbsVw8g8fgpUqTss2dW9WFXTtZTCpzFOmedRF6DrQfaaUFOEAohu2t6UKwygcFZqE3945xCBmSFYd3QbLpGMqVdEOkvWRXtJvj7eKom+/qt3OysR4UqM1Nj90mSEJZDRddFzNxWiq371FAbx7GEFoNVlxWGqf5K3g3T1fvEdPMR8gqLnlOO+8uXsio4oLHhrAU+KTIrKwztQ9I+cgdFzaM43m5Bfl0vPATHQfc4aGC6nKoXEcvdx7bbt29j2htOJorZREfQJHN9pbwa0cfO2OZEHSHLMmuJXPs1KS0uxIOH3xPd8C0SkqdkKrnG9t8pY00O7fC3RNs9aC8+Ino2ZLmIrVu2WHkAHDji6pXmjqFrWLe37Q/d0A20tWhsRaVHJ3bX9k60nrtrL240sgpTUBVYrwqO6IiFttQh7Qj6xx+i6WQrPOfPa2KsKS9UekVGRv6k1XcjYnv47/UNKlju/QJJSw8GBs3IrTawgnFhe9Nw02UGyGWyZ1s+/+zKquVBndQ/5tUPZxLqSkciHd5sDvBdoJk5480TwpSkewUNA1zRgmmOPePFthK1GSlJiZfpOYdJCI2Wx/Px8fkn9F+2v6S4a59qEF4JYittGcJ+nQrdOURHhvW+AE1539nZ8aWDq6srN0+fPt3Bzc3N0cnJycHFxcWxp880la0X5OW0HmgepK9JbF1Bf5758ZKJ+i4LVgQuVrB9Pp/vyHsdCwsPd+LmTSF7taZvsCBFYaXtQ2LFPVBqdOyp8dm+h4fH6wGdnZ0n8zM7v6Dw55PGMSj0o7bGlg6sX/Np/YtwHXj/x9575+3J21P9Fi1E8FI/pkzxX+f/BBhq9luuZDT4AAAAAElFTkSuQmCC","width":315,"height":325,"src":"/static/fff962ac6bfdb5c2156f04c8855b288b/b3029/thumbnail.png","srcSet":"/static/fff962ac6bfdb5c2156f04c8855b288b/b3029/thumbnail.png 1x,\n/static/fff962ac6bfdb5c2156f04c8855b288b/8d141/thumbnail.png 1.5x"}}},"authorName":"Core Services Team","authorDescription":"Core Services team is responsible for infrastructure and platform development at AUTO1","authorAvatar":null,"headerImage":{"relativePath":"pages/event-store-deadlock/header.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsSAAALEgHS3X78AAABoUlEQVQoz2MIjM+pDy/sfhxb3HkyNL/zXlbD9O+h2Y0XPD3c1RmAwM07gElKSooBhiUlxBlklDQYsAGDhIkMDP45Xf+t8hb9L5y66792/NT/wbUr/hTOOPDfN76whIEAMIjv5zRMmsJkmDiJH8QH0gwMftmdD21zZv8vmrT5l0v+7L/pXev/5XSv+hMUneLFzMCgr6ikrCApKckGdJ0QEAtKyynx8jEwcOiG1csZJk62B7rKDojVgIa5gm2JKWyf5V29+n9G19r/zkADg2pX/p+2bNtNDSVZM1ExcRNpaWkzoEFuQBwKxL5SUtJBEqLCrspmni7G6XMcDeInhAMNlDJMnJhpkDhRlCE6Jcc4pWrio/krNr4Pqlzw1zZz2r8J0+dddffwdAJZaGJqxg51HTcQqwANlBfhYmBTNvcWNEqdJQX0tiDQQFXDhEnyQFfKwoKDvTLJ28A9u/+jR/6M7xmemtIgwczMLCYGUkFxTSsjjB2bXbk3pahhE4i98/oXsLiysgoDcixLScswiAtwMQBdyGCUOpMR6EJQZDACXQnGAOs+imBvfQhTAAAAAElFTkSuQmCC","width":500,"height":168,"src":"/static/2d78359581a2f1861a243566e527e27a/7d852/header.png","srcSet":"/static/2d78359581a2f1861a243566e527e27a/7d852/header.png 1x"}}}},"html":"

TL;DR

\n

Always try to keep your transactions short and write your queries to deal with records in a deterministic order with respect to other transactions.

\n

Full Story

\n

Almost every non-trivial system needs some kind of task (job, event, whatever you call it) scheduling functionality at some point. Having more than 250 micro-services, ours was no exception.\nAt some point, it made sense to us to provide such functionality as a service for our developers. With event-store service, one simply can schedule a job to be executed at a specific time in future. There are already several ready to use tools for this specific purpose (e.g. BigBen, Quartz), however, because of several reasons (which are beyond the scope of this post) we decided to implement our own from scratch.

\n

Considering the subject of the post which is Deadlocks, we need a brief introduction on event-store's implementation.

\n

An event on the server side is simply a record in the database. A simplified view of events table is as follow:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
idevent_typedue_timestart_process_timeend_process_time
'5c55ea79-4774-49ae-96e2-975af7a41ce6''user-cache-flush''2019-01-01 00:00:00'nullnull
\n

The following is a bird's-eye view of what happens on the service:

\n

Thread #1
\na. The service periodically fetches a bunch of unprocessed events (those that are due and have no process time, neither start nor end) from the database (and marks them as progressing by filling the start_process_time column with the current timestamp). Let's call this fetch-unprocessed query.

\n

b. The service processes the events and marks them as processed (by filling the end_process_time column with the current timestamp). Let's call this save-processed query.

\n

Thread #2
\na. The service periodically checks the table and looks for stale events (those that are due and have start_process_time but no end_process_time). Let's call this fetch-stale query.

\n

b. The service marks them back as unprocessed (by setting start_process_time to null). Let's call this release-stale query.

\n

The very first MVP was a simple read-modify-write (anti-)pattern (with the help of Spring, JPA, and Hibernate). It is not hard to guess to what issue this implementation is vulnerable: Deadlocks.

\n
ERROR: deadlock detected\nDetail: Process 5234 waits for ShareLock on transaction 3465; blocked by process 467845.\n        Process 467845 waits for ShareLock on transaction 96575; blocked by process 5234.\nHint: See server log for query details.\nWhere: while updating tuple (14954,4) in relation \"events\"
\n
ERROR: deadlock detected\nDetail: Process 10438 waits for ExclusiveLock on tuple (14954,4) of relation 19118 of database 19113; blocked by process 31501.\n        Process 31501 waits for ShareLock on transaction 763124271; blocked by process 28450.\n        Process 28450 waits for ShareLock on transaction 763124277; blocked by process 28873.\n        Process 28873 waits for ExclusiveLock on tuple (14954,4) of relation 19118 of database 19113; blocked by process 10438.\nHint: See server log for query details.\nWhere: while locking tuple (6984,19) in relation \"events\"
\n

In the beginning it was not a big deal, because deadlock errors were not frequent and we could live with it. However, very soon, after getting more clients, the deadlock issue (among other problems, e.g. lost-updates) started hurting the quality of the service (e.g. firing events twice).

\n

For the first attempt, we refactored the code to get rid of read-modify-write antipattern:

\n
-- fetch-unprocessed query\nUPDATE events event_to_update\nSET start_process_time = now()\nFROM (\n    SELECT id\n    FROM events      \n    WHERE end_process_time is NULL\n            AND start_process_time IS NULL\n            AND (due_time BETWEEN ? AND ?)\n) matched_event_for_update\nWHERE event_to_update.id = matched_event_for_update.id\nRETURNING *;\n\n-- release-stale query (including fetch-stale query)\nUPDATE events event_to_release\nSET start_process_time = NULL\nFROM (\n    SELECT id\n    FROM events\n    WHERE end_process_time IS NULL AND start_process_time < ?\n) stuck_event\nWHERE event_to_release.id = stuck_event.id\nRETURNING *;
\n

Obviously this is an improvement, though reduced the number of deadlocks, it didn't solve the issue completely.\nThe quickest solution to eliminate deadlocks was to change the isolation level to Serializable. However this is the last solution we wanted since it would hurt the performance and the quality of the service (think what would happen if one service start a transaction, fetch the events to process, but for some reason become stale).

\n

The better way to deal with deadlock is to analyze transactions and queries to prevent transaction interference by:

\n
    \n
  1. making transactions short in terms of time
    2. \n
  2. writing queries to deal with records in a deterministic order
    3. \n
\n

By the first refactoring, we made transactions a bit shorter. But there's still some room for improvement. To apply the second best practice from the above we refactored the queries as follow (essentially ordering and acquiring a lock):

\n
-- fetch-unprocessed query\nUPDATE events event_to_update\nSET start_process_time = now()\nFROM (\n    SELECT id\n    FROM events      \n    WHERE ... -- as before\n    ORDER BY ID\n    FOR NO KEY UPDATE\n) matched_event_for_update\nWHERE ... -- as before\nRETURNING *;\n\n-- release-stale query (including fetch-stale query)\nUPDATE events event_to_release\nSET start_process_time = NULL\nFROM (\n    SELECT id\n    FROM events\n    WHERE ... -- as before\n    ORDER BY ID\n    FOR NO KEY UPDATE\n) stuck_event\nWHERE ... -- as before\nRETURNING *;
\n

However, the deadlocks were still appearing intermittently:

\n
ERROR: deadlock detected\nDetail: Process 7534 waits for ShareLock on transaction 93897000; blocked by process 13376.\n        Process 13376 waits for ShareLock on transaction 93896994; blocked by process 7534.\nHint: See server log for query details.\nWhere: while rechecking updated tuple (134131,3) in relation \"events\"
\n

Of course! we still had one transaction/query dealing with records in a non deterministic way: JpaRepository.save(Iterable<S> entities). In the code, after fetching events to be processed, we flag (UPDATE) the process events as processed, and also save (INSERT) new events spawned by the processed events. Here's the catch: the event processing is done in parallel. So the resulting list of events (both processed and newly created ones) is not ordered with respect to other transactions. So the third refactoring was to restore the order of processed events:

\n
/**\n * To process events.\n * The result list will be ordered with respect to the given list.\n * All new events will be added at the end of the result list.\n * @param events events to be processed\n * @return list of processed and created events\n */\nprivate List<Event> doProcessEvents(List<Event> events) {\n    ProcessingResult result = eventProcessor.processInParallel(events);\n\n    List<Event> successfulEvents = result.getSucceeded(); // may contain new events;\n    List<Event> failedEvents = result.getFailed();\n    List<Event> allEvents = ListUtils.union(successfulEvents, failedEvents);\n\n    Map<UUID, Integer> eventIndices = range(0, events.size()).boxed()\n            .collect(CollectorUtils.toMap(i -> events.get(i).getId(), i -> i));\n\n    // This should sort according to the original order and put all new events at the end.\n    allEvents.sort(comparingInt(event -> eventIndices.getOrDefault(event.getId(), Integer.MAX_VALUE)));\n\n    return allEvents;\n}
\n

After this last change, we saw the game for deadlocks was over: \"Goodbye Deadlock!\"

","fields":{"slug":"/event-store-deadlock/","tags":["deadlock","sql","transaction","postgresql"]}}},{"node":{"id":"7efdac28-b97f-53e0-adf8-48d022f3eecc","frontmatter":{"category":"Coding","title":"Integration Test Speedup with Spring","date":"2019-03-11","summary":"Here is what we found on speeding up your integration tests in spring based applications.","thumbnail":null,"authorName":"Boris Faniuk","authorDescription":"Boris is a former member of Engineering of AUTO1 Group.","authorAvatar":{"relativePath":"pages/integration-test-speedup/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/integration-test-speedup/header.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAPABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAABQABAv/EABYBAQEBAAAAAAAAAAAAAAAAAAIAAf/aAAwDAQACEAMQAAABJ6Uw4FJSv//EABsQAAICAwEAAAAAAAAAAAAAAAECAAMEFCEx/9oACAEBAAEFAl9tuawzH6WUPNYT/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQIBAT8BJ//EABsQAAEEAwAAAAAAAAAAAAAAAAABEBIhIjFR/9oACAEBAAY/AjJo9SizZ//EABwQAQEAAgIDAAAAAAAAAAAAAAERAHEQMUFR0f/aAAgBAQABPyFMdusqfAgeuBQwbbGHwF1jT4z/2gAMAwEAAgADAAAAELsf/8QAFhEBAQEAAAAAAAAAAAAAAAAAARBB/9oACAEDAQE/EA2f/8QAFxEAAwEAAAAAAAAAAAAAAAAAAAERIf/aAAgBAgEBPxDSpD//xAAbEAEBAQADAQEAAAAAAAAAAAABEQAhMXFBYf/aAAgBAQABPxAMiBZ241OQL4wZbyughIofpPEuZhdFS61EeZ//2Q==","width":1024,"height":768,"src":"/static/9f60dcc74069c89c91db86dc33946167/9f594/header.jpg","srcSet":"/static/9f60dcc74069c89c91db86dc33946167/9f594/header.jpg 1x"}}}},"html":"

Problem statement

\n

Over time we realized that our microservices built on top of Spring framework take tons of time on the integration test step. Every single PR runs all tests and this takes 10-20 minutes for some services. To decrease PR build time we decided to research why this happens.\nOne of the problems that this research highlighted is ramping up heavy spring context several times during execution of all module’s tests.

\n

As a proof of concept we have taken 3 modules and cleaned up their tests to achieve better integration test performance.

\n

The results are pretty much self-explanatory:

\n
    \n
  1. Module A now takes 3 minutes to test versus 11 before
    2. \n
  2. Module B now takes 3 minutes to test versus 8 before
    3. \n
  3. Module C now takes 4 minutes to test versus 12 before
    4. \n
\n

Below I list all factors that produce many spring contexts and how we managed to overcome them.

\n

Spring features to be avoided

\n

@ActiveProfiles and @ContextConfiguration with different attributes

\n

Our spring-based microservices actively use spring profiles and configuration classes.\nMany components from common libraries are hidden under profiles and configuration classes to not be active for every single module / environment without need. So, integration test authors define profiles and configuration classes that should be activated for their tests using these annotations:

\n
    \n
  1. @ActiveProfiles({\"integration-test\", \"pg-test\", \"mongo-test-two-nodes\"})
    2. \n
  2. @ContextConfiguration(classes = {Application.class, AwsTestConfiguration.class})
    3. \n
\n

Here mongo-test-two-nodes activates special mode for mongo based services to operate with two separate mongo instances. As not all tests need this, the profile is turned off by default and only those tests that need this behaviour would explicitly turn it on. But this, unfortunately, leads to multiple spring contexts being created.

\n

The same happens with AwsTestConfiguration that deploys special test-component to mock AWS calls.

\n

The solution was to make all tests use the same set of active profiles and context configuration classes for every single integration test.\nThe drawback is that some components are deployed, even for the tests that do not need it. But deploying single component takes O(1) time comparing with the whole spring context.

\n

Another point to keep in mind is that order of profiles (configuration classes) matters.\n@ActiveProfiles({\"integration-test\", \"pg-test\"})\n@ActiveProfiles({\"pg-test\", \"integration-test\"})

\n

Two settings mentioned above result in 2 different spring contexts.

\n

@SpyBean and @MockBean

\n

One interesting feature in spring-boot-test is mocking / spying real spring beans.\nThis was mostly used when we don’t care what some method does in test but we need to check that it was called by parent components.\nUnfortunately this also makes test to create separate spring context.\nWhat we’ve done here is replacing @SpyBean annotation with configuration class that wraps real spring bean into Mockito.spy(..):

\n
@Configuration\npublic class SomeComponentSpyFactory {\n\t@Autowired\n\tprivate SomeComponent someComponent;\n\n\t@Bean\n\t@Primary\n\tpublic SomeComponent someComponentSpy() {\n    \t\treturn Mockito.spy(someComponent);\n\t}\n}
\n

Another option is to use the combination of @Autowired and @Spy annotations.

\n
\t@InjectMocks\n\t@Autowired\n\tprivate ParentComponent parentComponent;\n\n\t@Spy\n\t@Autowired\n\tprivate ChildComponent childComponent;
\n

However, the drawback of the last solution is that, parentComponent needs to have a setter for childComponent which is not good in terms of encapsulation. But if you don’t use autowiring on constructor level it is fine. Anyway, one can choose whatever fits the situation better.

\n

@DirtiesContext

\n

One of the services used this annotation to clear the caches between different tests.\nClearing a cache is really a problem (see section below), but this solution comes to separate spring context per test class or even test method (depending on annotation parameters). Context per method was actually the case for a few test classes. Instead in this modules we added methods to clear the caches programmatically without deploying the whole new context.\nOne can use java reflection API instead of adding special production-level methods, again whatever fits you better.

\n

Challenge: test execution order and cleaning collections (caches)

\n

After all this “multiple context” factors are gone one may face (and we did) a challenge to fix or re-organize some tests. Before above enhancements were done all tests were running in separate spring contexts, therefore used separate instances of every component and therefore those problems could be hidden.

\n

As an example some tests in mongo-based service were testing getAll()-like operations. Those tests were running functions that fetched all records from some collections. This was not a problem as embedded mongo DB was different instance. After all tests started to exploit the same spring context some of them started to fail intermittently. The failures were intermittent because test execution order is different. One service may break another, but not vice versa.\nSo, to overcome this, we added missing @Before operations that cleaned those collections.

\n

The rule that I would suggest here is that every test should clean up whatever it needs to be clean and not rely on fairness of other tests. That was a problem in our case. Some tests had @After operations that cleaned up, but not everything produced by the test was cleaned up. This caused intermittent failures of other tests.

\n

Tools we created to reduce test execution time

\n

So, now after we know all this, we created two tools (and may create more in future) that help to identify those problems.

\n

Jenkins PR pipeline step

\n

After every PR commit we run one additional step that analyzes test execution log.\nIf more than one spring context is created this leaves some traces and a notification is sent to the author of the commit via slack

\n

As an enhancement, the jenkins jobs could be modified to fail the build if the number of spring contexts is greater than a certain threshold. The threshold could start with a high number and can gradually be reduced as the team cleans up the existing tests by adopting the aforementioned practices/fixes

\n

IntelliJ IDEA plugin

\n

We wrote an IntelliJ plugin to identify and solve the aforementioned problems. There will be a separate blog post about this plugin coming very soon.

","fields":{"slug":"/integration-test-speedup/","tags":["auto1","engineering","spring","java","testing"]}}},{"node":{"id":"99158851-0237-5904-98ca-02b5e2f45684","frontmatter":{"category":"Coding","title":"Spring 5 candidate component index case study","date":"2019-02-05","summary":"Analysis of Spring 5.X candidate component index applicability to boost our application startup time","thumbnail":{"relativePath":"pages/spring-5-indexer/thumbnailImage.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAZABQDASIAAhEBAxEB/8QAFwABAQEBAAAAAAAAAAAAAAAAAAIBBf/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAeBWySoZgAf/xAAaEAACAgMAAAAAAAAAAAAAAAAAARASICEx/9oACAEBAAEFAisLg95//8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAwEBPwEf/8QAFBEBAAAAAAAAAAAAAAAAAAAAIP/aAAgBAgEBPwEf/8QAFBABAAAAAAAAAAAAAAAAAAAAMP/aAAgBAQAGPwJP/8QAGhAAAwADAQAAAAAAAAAAAAAAAAERECAxIf/aAAgBAQABPyFK8Kx5obfaNVLt/9oADAMBAAIAAwAAABBTBzz/xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAEDAQE/EB//xAAUEQEAAAAAAAAAAAAAAAAAAAAg/9oACAECAQE/EB//xAAcEAEAAgMBAQEAAAAAAAAAAAABABEhMUFRYSD/2gAIAQEAAT8QR0FfCY14dprVxKUeSiVurchzX3sU1d7c6ZqWULr2CBDvz9f/2Q==","width":265,"height":325,"src":"/static/85788e7deb4eaccc84801ebd1f7b77a8/a2998/thumbnailImage.jpg","srcSet":"/static/85788e7deb4eaccc84801ebd1f7b77a8/a2998/thumbnailImage.jpg 1x,\n/static/85788e7deb4eaccc84801ebd1f7b77a8/99cce/thumbnailImage.jpg 1.5x"}}},"authorName":"Mariusz Sondecki","authorDescription":"Mariusz is Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/spring-5-indexer/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/spring-5-indexer/headerImage.jpg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAANABQDASIAAhEBAxEB/8QAFwAAAwEAAAAAAAAAAAAAAAAAAAECBv/EABQBAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhADEAAAAc/UgxB//8QAFBABAAAAAAAAAAAAAAAAAAAAIP/aAAgBAQABBQJf/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPwE//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPwE//8QAFBABAAAAAAAAAAAAAAAAAAAAIP/aAAgBAQAGPwJf/8QAGBAAAwEBAAAAAAAAAAAAAAAAAAERIEH/2gAIAQEAAT8hElKxzmP/2gAMAwEAAgADAAAAEPPP/8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAwEBPxA//8QAFBEBAAAAAAAAAAAAAAAAAAAAEP/aAAgBAgEBPxA//8QAGxAAAgIDAQAAAAAAAAAAAAAAAREAECExQVH/2gAIAQEAAT8Qm5fABBZ6Hh5TKTxX/9k=","width":1280,"height":853,"src":"/static/430c76d0d1e83aad3065e86e11366218/966a5/headerImage.jpg","srcSet":"/static/430c76d0d1e83aad3065e86e11366218/966a5/headerImage.jpg 1x,\n/static/430c76d0d1e83aad3065e86e11366218/aa36d/headerImage.jpg 1.5x"}}}},"html":"

The situation

\n

Since day one at AUTO1, we strongly believe that the microservice architecture helps us run business at scale not only due to the technical benefits it brings but also, if not foremost by allowing many teams to work independently from each other. Today, we’re running more than 250 microservices in production and on average add one more to our platform every week. All of them are powered by the Spring Framework technology stack, which we value very much for its productivity, but at the same time consider it to be a bit heavy in terms of bootstrapping time. The typical startup time of our Spring applications varies between 40 and 80 seconds. Since we do around 30 deployments per day, our engineering teams spend roughly 30 minutes daily waiting for the deployments to be completed. As we value the time of our engineering teams, we wanted to find a way to decrease these numbers.

\n

A new hope

\n

While investigating the startup times of our applications, we have identified classpath component scanning as a potential place to look for improvements. Luckily for us, some of our tech savvy engineers suggested a not so prominent Spring 5.X feature to come at help here - the build time component candidate index.\nThe build time component index is an alternative for regular component scanning, which skips the classpath search operation in favor of using a compile time pre-generated index of component candidates during the spring application context building. However, it comes with a small print from the Spring Framework experts saying that this functionality should have mostly a visible impact on startup times for applications with large amount of beans and that are operating in environments where IO operations are expensive (e.g. remote file systems) or where JVM security managers are in use. Although, neither of this is our use case, we have decided to proceed with the feature evaluation and measurements, as here at AUTO1, we like our decisions to be data driven.

\n

Demystifying the indexer

\n

The key part of the indexer feature is a pre-generated index file located in META-INF/spring.components one per JAR. Every index entry in this file is a fully qualified name of a candidate component as a key and comma separated stereotypes as value. So for example “X=Y, Z” can be read simply as register a candidate component X with following stereotypes Y, Z. Below is an example of a spring.components file:

\n
com.auto1.playground.service.SpringDummyService=org.springframework.stereotype.Component\ncom.auto1.playground.domain.SpringDummy=javax.persistence.Entity,javax.persistence.Table,javax.persistence.EntityListeners\ncom.auto1.playground.repository.SpringDummyRepository=org.springframework.stereotype.Component,org.springframework.data.repository.Repository\ncom.auto1.playground.configuration.LocalConfiguration=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomService=com.auto1.playground.service.IndexedCustomService\ncom.auto1.playground.Application=org.springframework.stereotype.Component\ncom.auto1.playground.SpringDummyController=org.springframework.stereotype.Component\ncom.auto1.playground.service.IndexedCustomExample=com.auto1.playground.annotation.IndexedCustom
\n

The first question that arises here is how to create such a file and keep it up-to-date. The Spring engineers thought about this problem as well and came up with an annotation processor tool called spring indexer that hooks in during the project build phase and generates the file. Making use of it is fairly simple, assuming you are using your favorite build automation tool, in our case maven and want our IDE to keep this file up to date, simply add the spring indexer annotation processor to your pom file:

\n
<build>\n    <plugins>\n        <plugin>\n            <groupId>org.apache.maven.plugins</groupId>\n            <artifactId>maven-compiler-plugin</artifactId>\n            <version>3.5</version>\n            <configuration>\n                <annotationProcessorPaths>\n                    <path>\n                        <groupId>org.springframework</groupId>\n                        <artifactId>spring-context-indexer</artifactId>\n                    </path>\n                </annotationProcessorPaths>\n            </configuration>\n        </plugin>\n    </plugins>\n</build>
\n

Once this is done, what happens during project build time is the spring context indexer (precisely speaking the CandidateComponentsIndexer class) while going through the project identifies annotated components and outputs them to META-INF/spring.components. The type of components it includes in the candidate list are:

\n
    \n
  1. \n

    Classes annotated directly or indirectly with spring @Component annotation (which by itself is meta annotated with @Indexed) and other stereotypes for which @Component is a meta annotation such as @Repository, @Controller, @Service, @Configuration.

    \n
    2. \n
  2. \n

    Classes and interfaces annotated with annotations from javax package, including CDI annotations (@Named, @ManagedBean), JPA annotations (@Entity, @EntityListeners) or even Servlet annotations (@WebListener)

    \n
    3. \n
  3. \n

    Any custom classes and interfaces annotated with @Indexed annotation.

    \n
    4. \n
\n

It should be noted that applying @Indexed on a class/interface where no other standard stereotype (as from points 1 and 2) is present will not make by itself the bean managed, thus not making them eligible for being autowired, but merely provide a hint that it represents a stereotype to be considered in the index.

\n

Example of the @Indexed annotation usage:

\n
@Indexed\npublic class IndexedCustomService { }
\n

Or as meta annotated as below:

\n
@Target(ElementType.TYPE)\n@Retention(RetentionPolicy.RUNTIME)\n@Documented\n@Indexed\npublic @interface IndexedCustom { }\n\n@IndexedCustom\npublic class IndexedCustomExample { }
\n

In order to make them available in the application context, thus autowirable, we could for example include them in the component scan filters as shown below:

\n
@SpringBootApplication\n@ComponentScan(basePackages= {\"com.auto1\"}, includeFilters = {\n  @ComponentScan.Filter(type= ANNOTATION, value = {IndexedCustomExample.class}),\n  @ComponentScan.Filter(type= ASSIGNABLE_TYPE, value = {IndexedCustomService.class})\n  }\n)\npublic class Application implements WebMvcConfigurer {\n    // ...\n}
\n

There is one big pitfall that comes with the feature though, that is by default the index is automatically enabled when a META-INF/spring.components file is found on the classpath during application start-up and this in turn, disables completely the regular classpath component scanning. In other words, both the index and classpath scanning cannot be used at the same time. The consequence of this is that if an index is available for some JARs, but is incomplete due to including other JARs that come with potential bean candidates but don’t have the index file generated, those beans will not be discovered, thus application startup might spectacularly fail with a well known error: “X required a bean of type 'Y’ that could not be found”.\nFortunately, Spring engineers, as thoughtful as they are, provided a feature flag spring.index.ignore that can be passed either as a system property or set in the spring.properties file at the root of the classpath allowing us to disable the usage of the candidate component index altogether.

\n

Test me one more time

\n

Like mentioned before, the aim of our investigation was to understand if the Spring indexer feature is of any help to us in regards to boosting the application start-up times. In order to make the test fairly reliable and control as many confounding factors as possible, we have decided to run the test firstly, using a “dummy” Spring 5 application (in two flavors - with 120 and 5000 beans) using all the features a production microservice would use, on a bare metal machine i.e. a MacBook Pro late 2017 i7. Secondly, running a real life production service in a less controlled but close to production environment i.e. on a r4.8xlarge EC2 instance with other services running concurrently. The applications themselves were using Spring 5.0.10.RELEASE with Spring Cloud Finchley.SR2 release train.\nAfter making a couple of hundred measurements of application bootstrapping times with candidate component index enabled / disabled and verifying in the logs that Spring was indeed using (or not) the component candidate list, we ended up with following averaged results:

\n

\n \n \n \n \n

\n

The expected disappointment

\n

As the above results have consistently shown, there was no improvement or even insignificant worsening (which will be a subject of our analysis in a separate article) of the bootstrapping time of the Spring applications, regardless of the number of candidate beans present, when using the pre-generated candidate component index. This was expected, as explained in the opening section, the conditions for which this feature was designed (slow or expensive IO operations) are simply not met in our case. One more important factor to consider here, is that a great deal of the beans in our Spring applications, is created using bean definition factory methods, which are not subject to the component classpath scanning, thus are not impacted by this feature.

\n

All in all, we deemed this feature not to be useful for our purposes and moved on to the next item on our list.

","fields":{"slug":"/spring-5-indexer/","tags":["spring","spring-indexer","bootstrapping","performance"]}}},{"node":{"id":"f7a13c08-6929-58c2-9d9e-eae3b357b1e1","frontmatter":{"category":"Coding","title":"Exploring technologies with Docker","date":"2018-12-19","summary":"Yet another use case for using containers.","thumbnail":{"relativePath":"pages/exploring-technologies-with-docker/thumbnailImage.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAZCAYAAAAxFw7TAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAHZklEQVQ4yx1VWVdaWRq9v6ueuh66U71WV7K6U9VVGYxG44SAmoCJiAoqYhRREZBJJkFmogIyyHABxQwaZZIZuXDh3ouVtiSzxtjXetjr2/vlO98+3zr7AGQzyT/govpJL0h+0hoxMOCkemmOwdRsmFOfATkI0zOGiSKiEkMdrdCUWXRYnUEGllPIoCKNXHO66lofIYNKnCviCNCiawVvLt0C/8G9Af59/gZ4W/qL7zb/l3Svqa9O1pGRu6K72GKEVxozFCut3EOMwI+jzZwDtJUbRbtx3s1P4PoQfTBzgD5TZFGAYiJ5hlwDnn5Lj5tk7HAPOalOpoOe4AZmUC44X530shH5a+Xxc8NRmaHJI+OruRo+WW1Uk62N6/I1uvKoSlenahPG49qYNlkDsBWS4+MazXlmppr/MDzWnJkHHBceVjTusxZH9NkKZy0HM7WJCk2RhPEG8KQ+j+sszNBco1BdcuznliKiChfkw1w/Hwa+aEihhrR158PyI+c3I9V6Jm3bPtcSM2nPOtS8kEG7eVGsdS6G4bYw3CbWvRjHHuLWcdtY80y8LvTEC0zPENoia8FGPSMY0FB22z/r+pwfV4imhpqo/qAhO75vDEcTbmuBqshVxlZzME1xBD+Tp2CGtohPWYSfLR/BQ6o0PKwqVAX2N1neNrcy6Z2CuSAXBk6XO2znRqr9k5a80VAR1j9ryesXDvbhqX22WndJKu/couo730p+ZXu1wI8oqsJtcWXeFoSm9MfwKN58xRLONPzz5Y8RGVx3z8HARw0peCptw9Eaaii6wv8TPQg09AOZ95bhk08vRtAzHaV+FRAei3cWjju1XfV+UyfCcYWqHXNJ7NepA8zkOCxc2ceQP3XPsHMbCwO+qNq9l7oe74Wm86/6baXdfbk+fHThYGGXTnbtwj6GXnp5Bfm2qMgGp1FOkFWdtQfhITmE9MmKiHljP3flmqpeOjnIVxsbAShTNgtTFLEwl8IWOj9kpfGCRr5uf1+xFqtKzAcliSVWUVnfZEWguKBOqMuSNxKIFxRD0pd6iB/G9ba8ZIgIodWIANKHBBBwkxUN/kh7E/xpdD/4A/Vl6Afqq8CdmWimg588uc+Jor9PR+vjpjL03MeBSXpS/ZGiAyXourHH5j60Vd6Ckk1krMfQj92RPEDHwSkUuDPx1n2HnXQ3TSfdt8di7n8zY5v9olwC3yBKkWSrTyR5hGMsFucDsxDbz6yNOAarTDe9OuUfrw7ZnlZZXkaVvcWo0jYo1YXtmSpAWnluYHqW9YMbAj3NJjAMb4q1sz7ta8UrW0UWWceXYYXkobXM/Fo4LwkkIbEvVhJuxUoivIr98b8g3IriNVGa39gtAR2G5tDvy/8J3eL/M3RTcCPcpPoveE/ya5Zi7T3pN/WhLfKm+mJkAWKsFuG22USdsHiEtczEsDZuArvm3bwk1sKJYg9nY9jAcgYDmkzdW7/p2jy35He3binv+u4Y2p1N2q7kkGu0znAOVvtMBFTyerHIMqTL/eLkdbLUSMJ4rU+crF1ziiyF9PAzNZIgV6MrCjUgvNprT/lZm2836daIlWKMuZm2fHguCgZ1ENf9siIGX8MLmxFoQpsoz1pLMNd6DM+YizgKOErwwouD4hJoq4h3XDDXaYKB7+qewFfZo9DFcrv9arXf+EXaHrzS9aZyrrXjR7w80ic4QrsWjtCOhRgeVTG0V5RAO3lRtH0Bx/wRtuQ+zE8FhhCitgNl+UZQ4P0K0f9eTQg21AT7R12/8UxFAC/MlFTcvXbculRAyNIs2ilM4SGRQnuFWXzzeZSIH0IUxhGSII3ynYeFcXCyRjRT0VEfGwVOZe2Wb9ZB62ddv+FMSVj9anhi+u6e3q85pdC23QHtu53lyKarIN3yFmSRYFm27YVmra9Kc9YSxLGWy3JLKB0LC0qpfVV5z78IAQ1Vj/9Peaf/dLk90FASwFNJ61bDSEt9eME4udgYRT6ZnmJX4GJBujNf7LeSscENYo3j9MMkfgptmouhRvt+/soxUftkGUW/2SYR4Lum3XehbPady5t831St/nPZ3a1L3PL5Gr2ON6x9MV83FBRlO0vFPhMFG1x7XJvdjMBE/H7vc9Ko2RbLXW1O1D5bGMilHX/LVt6sJbC6YvGpFRavWmnxKJcNu+sv9nCr8BvnZmnPtVl+aVvPKMKynD5nLGsSqpLAZzmWgC8hgT8CrQbsucTOUin5Sg7t+XgQ8K/xaOAGMx74eSIR+JF+AP6NHvU1L6SzJGn+3SN+uv6Ql/lj3AjBHN8cMuygn1DM1PrTDcoJw0WrPzET67TNZydUG/2k00Cus0KcE+C3yT3/zyOv/LfHX/tv0CKBn2i7vgczb9928aK5B9P76XtT+zn8Q9obc0/EiMaeXJuyLdOh6cyQjX2Z5uWWTI+BmCHpiJn7knuZcS8zAzRNhgO94nSAvJQMdC3G/QRBIkAUJn04vJ28mJcgTAWb2WFdr/7J2vTe8wADZHjHtse87F22dxQc9bJ2WV7WNh4RYYZ3yDnk/T90eFLdjx6IqwAAAABJRU5ErkJggg==","width":265,"height":325,"src":"/static/753d78469abf65ded853c2e0d4fb7aec/b3029/thumbnailImage.png","srcSet":"/static/753d78469abf65ded853c2e0d4fb7aec/b3029/thumbnailImage.png 1x,\n/static/753d78469abf65ded853c2e0d4fb7aec/8d141/thumbnailImage.png 1.5x"}}},"authorName":"Marcin Sałata","authorDescription":"Marcin is Senior Software Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/exploring-technologies-with-docker/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/exploring-technologies-with-docker/headerImage.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAGCAYAAADDl76dAAAACXBIWXMAAA7EAAAOxAGVKw4bAAABsUlEQVQY0zWR32/SUBiG+x8bM0IJMp2BjQ3SKgzTFlpWKG0puBloOefwQ5jbUDezC70wu9minO87JE6N/gPWYuLt+zwX7/t9kuacrpvtC9EgNjyhJeFRBX2mwg4rr+vkCDZs4/z3NtkOOxAdUgHTX6BpX6zr0RE8HZexTRQhxWQPf4Q6WCdDyHofYD42VqcT42vWu8RNdj80ICZFjOkufg81ME/Cf96CNlfxpMB/hi+weTyAR/41TEZNLrlsFytkHzPdN5Bu3IpMdwnp3lv+wPgittz3UKMl6LACBqSEhyMVMv4ZbJk3Qu4vwKEKV6PnmPLeYca6wYfdBUra9ADLkyLk+jOQzVvMutdQda94rXUlDNJa5V8X4ZBVMNWbY7r1CbdfTrkyqqBFNFCDBa+6c2FQe5VnNTAjA6WY5fFbMkXrjyHrfkSawD/jIv891NF5dQy57mUyTwdKrIR/BqPP4D6qQjzO8zhp+GvQwPbGC5YwpwaXnA4TPWcmbGLxXKhgMHoGEStwPdxH2TvHVP1OPA6m6EYqbkcK2smdgvZMuD7FAd0DPSwK2VtiunEn5ORJfwFIJFacA7WGDwAAAABJRU5ErkJggg==","width":1280,"height":392,"src":"/static/be0acc48b4db44e7e3d00e2d5c353dc3/26421/headerImage.png","srcSet":"/static/be0acc48b4db44e7e3d00e2d5c353dc3/26421/headerImage.png 1x,\n/static/be0acc48b4db44e7e3d00e2d5c353dc3/34a7a/headerImage.png 1.5x"}}}},"html":"

tl;dr

\n

Using Docker Compose we are able to quickly set up complex environments consisting of multiple containers, which allows to speed up software evaluation process.

\n

Intro

\n

Things evolving quickly, tools come and go, last year cutting edge is often legacy now. Dozens of new tools emerges every month\nand approach for being up to date also needs to evolve.

\n

Containers on production with all its benefits and challenges is a common thing nowadays, but do not forget that containers could also help us in other use cases. One of them is to improve technologies evaluation process.

\n

Docker at home

\n

Many moons ago when new fancy tool was released and looks so promising to give it a try options were limited - build it from sources, use installer or package manager. Giving a try for new promising tool was quite time consuming. Steps required to download, install, configure system and dependencies were usually not most exciting part of evaluating new message broker or database. Now containers or PaaS solutions allow us to shift focus directly from setup to features that particular products offers.

\n

Level 1: Basic usage

\n

Using Docker one terminal command is enough to download, run and evaluate interesting applications.

\n
docker run postgres
\n

Above example command will pull latest version of PostgreSQL image from Docker Hub and run it locally. This is a quick and straightforward way of running hundreds of interesting applications. Container will run in sandbox environment, so we do not need to think about operation system settings and other specific configuration that may be required with standard installation. Although it is a simpler way than using installers or building from sources, then comparing to common package managers where also one command is usually enough, user experience looks quite similar.

\n

Level 2: Next step

\n

The real benefits of using Docker in local environment we can observe if we would like to build some complex environment that contains multiple applications linked together as usually things are more complicated than just running a database or monitoring tool separately. Challenge often starts when we would like to evaluate a monitoring tool that needs a datasource and entire environment should be relatively easily to set up and manage. That is where Docker Compose comes to help.

\n

As stated in official documentation Compose is a tool that allow us to define and run multi-container applications. Using it in most common scenarios comes down to creating YAML file with required images and configurations.

\n
version: '3'\nservices:\n  myService1:\n    image: service1\n    {configuration:}\n  myService2:\n    image: service2\n    {configuration:}
\n

Having docker-compose.yml file ready we are able to start it with one terminal command.

\n
docker-compose up
\n

Docker will build or download all necessary images, set up necessary configurations, networks and run defined containers for us.

\n

Level 3: Hands-on experience

\n

In below example we would like to see in action Grafana - open source monitoring tool with PostgreSQL as datasource.

\n

Define PostgreSQL

\n

First thing w need to do is to create empty docker-compose.yml file and add PostgreSQL as datasource.

\n
version: '3'\nservices:\n  myDatasource:\n    image: postgres
\n

Above we have defined one service called myDatasource that will be created using postgres image.

\n

Define Grafana

\n

Having our datasource configured we can extend our Compose file with second service - Grafana.

\n
version: '3'\nservices:\n  myDatasource:\n    image: postgres\n  myMonitoring:\n    image: grafana/grafana\n    ports:\n      - 3000:3000
\n

Above we have defined second service called myMonitoring that will be created using grafana/grafana image.\nAdditionaly to basic service definition we have also exposed port 3000, so Grafana user interface can be access from outside the container.

\n

Troubleshooting

\n

At this point we have configured our environment, so by running docker-compose up two containers should be configured and run by Docker.\nThen navigating to localhost:3000 will allow us to access Grafana web interface where datasource could be set up.\nThing that is worth notice is that if we would like to configure our PostgreSQL datasource with default suggested Host - localhost:5432

\n

\n \n \n

  • the first commit dates back to 1st of Aug, 2013, by the current VP of Engineering
    • \n
  • it was built on the Silex framework, using ORM (doctrine) with direct connection to MySQL
    • \n
  • 32.5k commits
    • \n
  • ~100 contributors
    • \n
  • ~600k lines of code
    • \n
  • at some point in time, we had 4 teams dedicated only to Admin
    • \n\n

    From these facts, you can see, that what the thing we’re talking about - it's a typical monolith.

    \n

    An authentication between requests was handled with a very common approach for PHP applications - PHP Sessions. With this approach the session identifier for the current user is put in a cookie, and on the server side, the session data is somehow stored as a document.

    \n

    \"Monolith

    \n

    The moment any monolith is afraid of

    \n

    Then we’ve got our first microservices. Some business logic started to live its own life. And of course, our Admin should have had some control over it. So our Admin started to interact with our microservices, and initially it was treated more or less as a third-party API.

    \n

    \"First

    \n

    Our microservices are located behind an API gateway (we call it GateKeeper), every authenticated request should be signed with an OAuth token to get through it. Also, every request should be signed by the client application with an API key.

    \n

    Admin has become one of the external clients of microservices API. It had its own API key, every user was obtaining an OAuth token, which is kept in a session document, as a part of a session data.

    \n

    Months were passing. In general, an architecture stayed the same, apart from the fact the number of our microservices increased. And increased a lot.

    \n

    \"More

    \n

    More and more business logic was covered by services while less and less logic was implemented in Admin. Then at some point were realized, that what happens in Admin - it’s just a proxying of requests to a gateway. The usual code for a new feature was more or less - send a request to the Gateway, get a request, send this request to a front-end client. Not the most exciting engineering work ever.

    \n

    Moreover, we’ve got another complication - some teams without any PHP engineer had to add some new features to back-office. For this, one needed to create a proxying code, and for this one would need some support from another team.

    \n

    There should be a better way

    \n

    So we’ve started to think of possibilities to send requests to the Gateway directly from Frontend, in order to avoid a necessity to make any implementation in Admin every time we do a new feature.\nAnd challenges were:

    \n\n

    Then unexpectedly a group of our engineers (Mikhail, Oleg, Alexey and Andy - thank you, guys!) came out with a prototype of Vigolante, as a Hackathons project.

    \n

    Vigolante - a mix of GO and VIGILANTE (sp.), which means a guard.

    \n

    \"Vigolante\"

    \n

    The idea behind it can fit in one sentence: it’s an authentication layer, which uses the same cookie as Admin, gets a session identifier from a frontend request, connects to the sessions storage (Memcached), fetches a session data, gets a OAuth token from the data and, enriches the initial request with the token and then forwards it to a Gateway.

    \n

    Quite a simple flow, but once in place helped us to save a lot of development time in order to deliver new features to Admin. From now on, we don’t need to make any changes in Admin, when we need to send a request to a microservice from Frontend.

    \n

    We have chosen Golang as an underlining technology to build the project. The reasons are:

    \n\n

    Basically, the project was developed by the efforts of 4 engineers in a bit more than one day. We are proud to say, that this time was not just for coding but also for taking care of bringing the software to production; things as:

    \n\n

    There were few things yet to be resolved (such an implementation of mechanisms of refreshing of OAuth tokens, also there were some problems with encryption of PHP sessions from Memcached), but those were successfully resolved; one month later we released the first feature relying on Vigolante to production.

    \n

    Feeling proud and fulfilled, we could now focus our attention on the our engineering challenges.

    ","fields":{"slug":"/vigolante/","tags":["golang","microservices","monolith","PHP","back-office"]}}},{"node":{"id":"f1483025-8f93-596d-89a4-e459c4718414","frontmatter":{"category":"Coding","title":"Database Transaction Isolation","date":"2018-06-20","summary":"What is transaction isolation level? What are differences? What are the default values for RDBMSs?","thumbnail":{"relativePath":"pages/transactionisolation/thumbnail.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAABIUlEQVQoz5VS7U7DMAzsa6A1cRynTT/WrTBNiB8gJN7/mczFVWHTWiF+nM750OXsS8XMGjhqbDplaRZuMupWKSZ1zqn3/g5EtIsqBNIAoTRdNeajpuPFIMOs0p+0Pjypqw9AvbADQ7Q8tAUIBg1wwk0PwdGEuO2VUTPcynAGnrE3GlNsFnHnH5wXVGvLQbLmy4ehe/3SfP20OnWTpmHQbh41n9BB32IvquS4OQ4TpLIgj7mhDh6OyyPB1t47c1TYu4UJYyr3dwXtgGpNmSFCgDdOHcbBZO3dw/2I3da/Dlm0nd8ww0ljf14CAsv4gofCppNV7MGhoD1uB8zrHa2wfZUSRuGSvve0K7gZSkrpJvZ6wfo9LE2362ZTUET+vPwfh9/1JBQjnS3wvwAAAABJRU5ErkJggg==","width":677,"height":325,"src":"/static/2691083d1efbcb8a271d80bf9ff6940f/b3029/thumbnail.png","srcSet":"/static/2691083d1efbcb8a271d80bf9ff6940f/b3029/thumbnail.png 1x"}}},"authorName":"Hayk Jhangiryan","authorDescription":"Hayk is Senior Java Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/transactionisolation/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/transactionisolation/header-image.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAABIUlEQVQoz5VS7U7DMAzsa6A1cRynTT/WrTBNiB8gJN7/mczFVWHTWiF+nM750OXsS8XMGjhqbDplaRZuMupWKSZ1zqn3/g5EtIsqBNIAoTRdNeajpuPFIMOs0p+0Pjypqw9AvbADQ7Q8tAUIBg1wwk0PwdGEuO2VUTPcynAGnrE3GlNsFnHnH5wXVGvLQbLmy4ehe/3SfP20OnWTpmHQbh41n9BB32IvquS4OQ4TpLIgj7mhDh6OyyPB1t47c1TYu4UJYyr3dwXtgGpNmSFCgDdOHcbBZO3dw/2I3da/Dlm0nd8ww0ljf14CAsv4gofCppNV7MGhoD1uB8zrHa2wfZUSRuGSvve0K7gZSkrpJvZ6wfo9LE2362ZTUET+vPwfh9/1JBQjnS3wvwAAAABJRU5ErkJggg==","width":838,"height":402,"src":"/static/2691083d1efbcb8a271d80bf9ff6940f/d5ccc/header-image.png","srcSet":"/static/2691083d1efbcb8a271d80bf9ff6940f/d5ccc/header-image.png 1x"}}}},"html":"

    Database Transaction Isolation

    \n

    What is transaction isolation level?

    \n

    Transaction isolation level defines the degree to which one transaction must be isolated from resource or data modifications made by other transactions. 

    \n

    Let's have a look at phenomena which can occur during the execution of concurrent transactions:

    \n\n

    Isolation is the I in the acronym ACID. In ACID, atomicity and durability are strict requirements. Whereas consistency and isolation are more of configuration (kind of). Besides, they are closely related, so much so that the SQL standard defines four levels of transaction isolation based on the consistency they provide (from least to the most consistency):

    \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    ISOLATION LEVELDIRTY READNON-REPEATABLE READPHANTOM READ
    READ UNCOMMITTEDpossiblepossiblepossible
    READ COMMITTED--possiblepossible
    REPEATABLE READ----possible
    SERIALIZABLE------
    \n

    Different isolation levels are defined based on whether they allow (or prevent) the above phenomena (as described by the SQL standard). Note that transaction isolation does not affect the changes made by the same transaction. It means a transaction always sees all the changes made by itself.

    \n

    Now what?

    \n

    Inside Java services, the transaction isolation level can be controlled by Spring @Transactional annotation which supports isolation attribute. If you do not specify the isolation level in the code explicitly, the default isolation level will be applied, which means that the default isolation level of the underlying datastore will be used.

    \n

    It is worth to mention that default isolation levels for each system might be different. For example, the default isolation level for MySQL 5.7 (InnoDB) and PostgreSQL 9.5 is respectively, REPEATABLE READ and READ COMMITTED. To check the default isolation level, query:

    \n\n

    References

    \n
      \n
    1. MySQL Transaction Isolation Levels
      2. \n
    2. Transaction Isolation in PostgreSQL
      3. \n
    3. Useful article
      4. \n
    ","fields":{"slug":"/transactionisolation/","tags":["coding","sql","transactions"]}}},{"node":{"id":"c61acf9d-7746-503a-ba8f-cf96f4042096","frontmatter":{"category":"Coding","title":"Secure coding practices for NodeJS Web Applications","date":"2018-06-20","summary":"This post highlights various coding practices for securing a NodeJS web application against the most critical web attacks.","thumbnail":{"relativePath":"pages/securing-nodejs-applications/thumbnail.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFwABAAMAAAAAAAAAAAAAAAAAAAECBf/EABUBAQEAAAAAAAAAAAAAAAAAAAEC/9oADAMBAAIQAxAAAAHMkZoCv//EABUQAQEAAAAAAAAAAAAAAAAAAAAS/9oACAEBAAEFApSl/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BJ//EABURAQEAAAAAAAAAAAAAAAAAAAEQ/9oACAECAQE/AWf/xAAXEAADAQAAAAAAAAAAAAAAAAAAECEx/9oACAEBAAY/AoYv/8QAGhABAAIDAQAAAAAAAAAAAAAAAQARIUFxUf/aAAgBAQABPyHOqLFGlciV85P/2gAMAwEAAgADAAAAEHvP/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAh/9oACAEDAQE/EAJs/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFjP/8QAHBABAAEEAwAAAAAAAAAAAAAAASEAETFxUWHw/9oACAEBAAE/EMpW8q+4qwMgymddUsU0EV//2Q==","width":867,"height":325,"src":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/a2998/thumbnail.jpeg","srcSet":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/a2998/thumbnail.jpeg 1x"}}},"authorName":"Rajababu Pradhan","authorDescription":"Raja is Senior Frontend Engineer at AUTO1 Group.","authorAvatar":{"relativePath":"pages/securing-nodejs-applications/avatar.png","childImageSharp":{"resolutions":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAACXBIWXMAAAsSAAALEgHS3X78AAACyElEQVQ4y4VUO2gUURR9mZmN8YcItoKVjeXuzsxu9jOzm92NkBBTqDEmjZAEgtqIpSgI1gp+uhS6kEI7wc7GQsFCLERiISGCaGHhF02yu+M5s/cujyC4cJY379573r3n3XdNvV51xsYiJ4rKbqlUyBj8wjAYyWazp4BVYB3YAraBDeAhMBcE/l76lkrFTLVachuNyCGXEbJMrVbx6ADnaWAtn88nvu8nWBMdQcJ9Auv3wCxjWq2aC9IMSF3DzCyyG1YAs9pSYkL2melmLpfT71tKGkUlz2iZJAuCgA5dEjGAwPoV0AbuAy+tfR7YkZjb5CiXR1OutEw5jWQdCfgCnEiSxNg/7LWAj+KTSiHSzBvrAtascrriPCmnDkP4AYQ0Em3TGPHf8P38ARpP/4PsHQ7aU69Hqdi4OJeI47JXLg9IX1tVbUuWCzSsygcJO+L0goGNRpySoB1colarerhEvcCn4tuxCB/TsC5ZdS3Ct2Ho75uZmRoigWbIbgCxt7Q0O8TLsgg17o2R2+JHzzJcV/1IoBlybel42SLsCcc3I6UqoaZ+lUHFYjjCzJSQa+4J4UVLKiX8ScOGVXJXDN+xd7SfZXFYS65U+n2Wz+cOw+ezHadSkfCRdVJiZdlmMG9ZtHMpgWR3Z0eMSvWExjmrbXoCbZ951dIim7Yk6u1IYtlwavChW51Pp01tXOwf01eC9RHs/Rbbnx2Hf4IUh/Q5zdol8H3KG22Pj9edOK54LF1876pd37TELhqdZ+J4MwxDHUvXgDOFQrB7YqLpjI4WdhG8GLYNbCeBK/1+TWNWRG+Pfy5Hj5Dek3IXzX9+8DkL/AIeLC/PDU1NHXdSQk5altNsDkjPAT+ArzKyFnK5bB2IxbYik4hanmfM5GTL6Q/Y2DUc25y0HLToubR8TI39cL4APJOxr/35AXgOXEJnHKQvepP6eiQj119kDrZjqLo8HAAAAABJRU5ErkJggg==","width":50,"height":50,"src":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png","srcSet":"/static/26a7a327ccc4b335712e5a7086f2b26d/45876/avatar.png 1x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/eb85b/avatar.png 1.5x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/4f71c/avatar.png 2x,\n/static/26a7a327ccc4b335712e5a7086f2b26d/9ec3e/avatar.png 3x"}}},"headerImage":{"relativePath":"pages/securing-nodejs-applications/header-image.jpeg","childImageSharp":{"resolutions":{"base64":"data:image/jpeg;base64,/9j/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wgARCAAIABQDASIAAhEBAxEB/8QAFwABAAMAAAAAAAAAAAAAAAAAAAECBf/EABUBAQEAAAAAAAAAAAAAAAAAAAEC/9oADAMBAAIQAxAAAAHMkZoCv//EABUQAQEAAAAAAAAAAAAAAAAAAAAS/9oACAEBAAEFApSl/8QAFREBAQAAAAAAAAAAAAAAAAAAARD/2gAIAQMBAT8BJ//EABURAQEAAAAAAAAAAAAAAAAAAAEQ/9oACAECAQE/AWf/xAAXEAADAQAAAAAAAAAAAAAAAAAAECEx/9oACAEBAAY/AoYv/8QAGhABAAIDAQAAAAAAAAAAAAAAAQARIUFxUf/aAAgBAQABPyHOqLFGlciV85P/2gAMAwEAAgADAAAAEHvP/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAh/9oACAEDAQE/EAJs/8QAFhEBAQEAAAAAAAAAAAAAAAAAARAR/9oACAECAQE/EFjP/8QAHBABAAEEAwAAAAAAAAAAAAAAASEAETFxUWHw/9oACAEBAAE/EMpW8q+4qwMgymddUsU0EV//2Q==","width":1200,"height":450,"src":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/935ac/header-image.jpeg","srcSet":"/static/f21842b8ed2aa0a1b2cd57f6dbf9e7fc/935ac/header-image.jpeg 1x"}}}},"html":"

    Secure coding practices for NodeJS Web Applications

    \n

    Everyone would agree that security is very important and with the increase in cyber attacks in the recent past, organizations are investing\nheavily on application security. In this post lets talk about securing a web application built in NodeJS.

    \n

    NOTE: The security concepts discussed here are language or framework agnostic. However, here we will see how these practices are implemented\nin NodeJS web applications.

    \n

    1. Use SSL/TLS for communication

    \n

    It is always a good practice to send your data over HTTPS rather than HTTP and it is imperative if your app transmits sensitive data.\nEncrypting data transmitted between the client and server helps mitigate several attacks like man-in-the-middle(MITM) attack, packet sniffing,\neavesdropping etc. Let’s see how to set up TLS/SSL in Express 4.x:

    \n

    Lets first generate a self-signed certificate:

    \n
    $ openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365
    \n

    This generates a self-signed certificate valid for 365 days.

    \n

    NOTE: The self-signed certificate is not ideal for production. For production, you should get a certificate from a Certificate Authority(CA).

    \n

    Next, enable HTTPS on Express. Additionally, redirect all HTTP traffic to HTTPS:

    \n
    const fs = require('fs');\nconst https = require('https');\nconst express = require('express');\n\nconst NODE_ENV = process.env.NODE_ENV || 'development';\nconst PORT = process.env.PORT || 3443;\n\nconst app = express();\n\nhttps.createServer({\n  key: fs.readFileSync('/path/to/key.pem'),\n  cert: fs.readFileSync('/path/to/cert.pem')\n}, app).listen(PORT);\n\n// Redirect http requests to use https in production\nif (NODE_ENV === 'production') {\n  app.use((req, res, next) => {\n    if (req.header('x-forwarded-proto') !== 'https') {\n      res.redirect(`https://${req.header('host')}${req.url}`);\n    } else {\n      next();\n    }\n  });\n}
    \n

    2. Use security headers generously

    \n

    i) Strict-Transport-Security: The HTTP Strict Transport Security(HSTS) if set in the response header, tells the browser that it should only communicate using HTTPS instead of HTTP while communicating with the specified domain.

    \n

    Syntax:

    \n

    Strict-Transport-Security: max-age=<expire-time>

    \n

    Here, max-age is the time(in secs) that the browser should remember that this site is only to be accessed using HTTPS.

    \n

    Example: from facebook.com:

    \n

    strict-transport-security:max-age=15552000;

    \n

    ii) X-Frame-Options: This HTTP response header can be used to indicate whether or not a browser should be allowed to render a page in a <frame>, <iframe> or <object>. Sites can use this to avoid clickjacking attacks, by ensuring that their content is not embedded into other sites.

    \n

    Syntax:

    \n
    X-Frame-Options: DENY\nX-Frame-Options: SAMEORIGIN\nX-Frame-Options: ALLOW-FROM https://siteutrust.com/
    \n

    iii) X-XSS-Protection: This HTTP response header enables the built-in XSS filter in modern browsers.

    \n

    Example:

    \n

    X-XSS-Protection: 1

    \n

    iv) X-Content-Type-Options: This response HTTP header is a marker used by the server to indicate that the MIME types advertised in the Content-Type headers should not be changed and be followed. This prevents MIME type sniffing attacks.

    \n

    Syntax:

    \n

    X-Content-Type-Options: nosniff

    \n

    v) Content-Security-Policy: Prevents a range of injection attacks including Cross Site Scripting(XSS) attack.

    \n

    Syntax:

    \n

    Content-Security-Policy: policy

    \n

    For a detailed explanation of CSP, go through this link.

    \n

    To set these headers in NodeJS, use the helmet npm package:

    \n
    const express = require('express');\nconst helmet = require('helmet');\n\nconst app = express();\n\n<b>app.use(helmet())</b>
    \n

    This sets all the necessary headers in response.

    \n

    To set the headers individually:

    \n
    app.use(helmet({\n  frameguard: {\n    action: 'deny'\n  }\n}));
    \n

    For an exhaustive list of security headers that should be set in a web application, check out the OWASP Secure Headers Project.

    \n

    NOTE: In some web servers, the security headers can be set in the server configuration file itself. For example, in nginx server, we can set the above headers in nginx.conf as shown below:

    \n
    add_header X-Frame-Options DENY;\nadd_header X-Content-Type-Options nosniff;\nadd_header X-XSS-Protection 1;\nadd_header Content-Security-Policy \"default-src 'self'\";
    \n

    3. Preventing CSRF attacks

    \n

    Cross site request forgery (CSRF), also known as XSRF, Sea Surf or Session Riding, is an attack vector that tricks a web browser into executing an unwanted action in an application to which a user is logged in. CSRF attacks specially targets state-changing requests and can force the victim to transfer funds, change email/password and so on.

    \n

    CSRFs are typically conducted using social engineering, such as an email or link that tricks the victim into sending a request to a server on behalf of the attacker. The server has no way to distinguish a forged request from a genuine one.

    \n

    In NodeJS, to prevent CSRF attack, we usually use the csurf express middleware:

    \n
    const cookieParser = require('cookie-parser');\nconst csrf = require('csurf');\nconst bodyParser = require('body-parser');\nconst express = require('express');\n\nconst csrfProtection = csrf({ cookie: true });\nconst parseForm = bodyParser.urlencoded({ extended: false });\n\n// create express app\nconst app = express();\n\n// we need this because \"cookie\" is true in csrfProtection\napp.use(cookieParser());\n\napp.get('/form', csrfProtection, (req, res) => {\n  // pass the csrfToken to the view\n  res.render('send', { csrfToken: req.csrfToken() });\n});
    \n

    In the view use the CSRF token passed:

    \n
    <form action=\"/process\" method=\"POST\">\n  <input type=\"hidden\" name=\"csrf_token\" value=\"{{csrfToken}}\">\n\n  Enter amount: <input type=\"number\" name=\"amount\">\n  <button type=\"submit\">Submit</button>\n\n</form>
    \n

    4. Preventing XSS attacks

    \n

    Cross-Site Scripting (XSS) attacks are a type of injection, in which malicious scripts are injected into otherwise benign and trusted web sites.\nAn attacker can use XSS to send a malicious script to an unsuspecting user. The end user’s browser has no way to know that the script should not\nbe trusted, and will execute the script. Because it thinks the script came from a trusted source, the malicious script can access any cookies, session tokens, or other sensitive information retained by the browser and used with that site. These scripts can even rewrite the content of the HTML page.

    \n

    The thumb rule to prevent this category of attack is to always validate and sanitize user data before processing or storing in database. Never trust data coming from user.\nValidation must be done on the server-side as client-side validation can be easily bypassed using tools such as Burp Suite, TamperData etc.

    \n

    A common approach to validate and sanitize user data is to use a library like validator.js.

    \n

    Example: To validate an email

    \n
    import validator from 'validator';\n\nif(validator.isEmail('foo@bar.com')) {\n  // Process email or store in DB\n}
    \n

    This library provides a number of validators and sanitizers to filter user inputs.

    \n

    Other useful libraries include DOMPurify and xss-filters.

    \n

    Here is an example to sanitize user input using xss-filters:

    \n
    const express = require('express');\nconst app = express();\nconst xssFilters = require('xss-filters');\n\napp.get('/', (req, res) => {\n  let firstname = req.query.firstname; //an untrusted input\n  res.send('<h1> Hello, ' + xssFilters.inHTMLData(firstname) + '!</h1>');\n});\n\napp.listen(3000);
    \n

    5. Preventing SQL Injection(SQLi) attacks

    \n

    Passing unvalidated user input directly to a SQL statement is vulnerable to SQL injection attack.

    \n

    Conside the following example:

    \n
    // SQL query vulnerable to SQLi\nsql = \"SELECT * FROM users WHERE username='\" + username + \"' AND password='\" + password + \"'\";\n\n// Execute the SQL statement\ndatabase.execute(sql)
    \n

    Now suppose the user enters the following in the username field:

    \n

    ' OR '1'='1' --

    \n

    The above SQL statement becomes:

    \n
    sql = \"SELECT * FROM users WHERE username='\" + ' OR 1=1 -- + \"' AND password='\" + password + \"'\";
    \n

    This effectively nullifies the need of a password and returns all the users in the database.

    \n

    This attack can be completely prevented by using parametrized or prepared statement.

    \n

    If you’re using an ORM to access the database (Mongoose, Sequelize etc), the ORM will normally take care of SQL injection by using prepared statements under the hood.

    \n

    6. Secure cookies using cookie flags

    \n

    XSS vulnerability in an application can be used to steal browser cookies. To prevent cookie stealing we can set the httpOnly flag of the cookie.\nAdditionaly, we can tell the browser to send cookies only over HTTPS using the secure flag.

    \n

    secure : this attribute tells the browser to only send the cookie if the request is being sent over HTTPS.

    \n

    HttpOnly : this attribute is used to help prevent attacks such as cross-site scripting, since it does not allow the cookie to be accessed via JavaScript.

    \n

    Example:

    \n
    app.use(session({\n  secret: ‘My super secret’,\n  <b>cookie: { httpOnly: true, secure: true }</b>\n}));
    \n

    7. Preventing brute force and DoS attack

    \n

    To prevent our site from overwhelming with a large number of requests, we need to put some kind of rate limiting to our API.

    \n

    We can use the ratelimiter npm package to implement rate limiting. If you are using Express, the express-rate-limit middleware can be used as shown below:

    \n
    const RateLimit = require('express-rate-limit');\n\nconst limiter = new RateLimit({\n  windowMs: 15*60*1000, // 15 minutes\n  max: 100, // limit each IP to 100 requests per windowMs\n  delayMs: 0 // disable delaying — full speed until the max limit is reached\n});\n\n// apply to all requests\napp.use(limiter);
    \n

    8. Error Handling

    \n

    Any error in the application should be handled gracefully by showing a custom error page to the user instead of showing stack trace in the error page\nthereby leaking sensitive infrastructure information like server info.

    \n

    9. Tools of the trade

    \n

    i) The Node Security Project\nTo check the various npm modules for known vulnerabilities, the Node Security Project provides the nsp tool to check for vulnerabilities:

    \n
    $ nsp check
    \n

    ii) Synk\nSynk checks the application against Snyk’s open source vulnerability database for any known vulnerabilities in our dependencies.

    \n
    $ npm install -g snyk\n$ cd your-app\n$ snyk test
    \n

    iii) nmap\nNmap (“Network Mapper”) is a free and open source utility for network exploration or security auditing.

    \n

    iv) sqlmap\nsqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers.

    \n

    v) Burp Suite\nThe must-have tool for application penetration testing. It includes an automated scanner to detect most common vulnerabilities in a web application.

    \n

    Further Readings

    \n

    OWASP Top 10 Most Critical Web Application Security Risks

    \n

    NodeJS Security Checklist

    ","fields":{"slug":"/securing-nodejs-applications/","tags":["nodejs","javascript","security","appsec","infosec"]}}}]}},"pageContext":{"slug":"/categories/Coding","category":"Coding","categories":["Architecture","Coding","DevOps","Engineering","ProjectManagement","QA","Social","TechRadar"]}}