DSL.using(java)
   .toGoBeyond(BeanValidation)
   .at(Snowcamp.eq(fr));

Welcome to the Furets!

@dubreuia – Alexandre Dubreuil

  • French canadian working in Paris since 2009
  • Software Architect at LesFurets.com


@gdigugli – Gilles Di Guglielmo

  • Designer of sweet cooked software since 1999
  • Software Architect at LesFurets.com

  • 1 website, 5 Insurance Products : Car, Health, Home, Bike, Loan
  • 1 codebase, 450k lines of code, 60k unit tests, 150 selenium tests
  • 22 Developers, 2 DevOps, 4 Architects
  • 19 production servers including Load balancers, Frontend, Backend, Databases, BI
  • 1 release per day
  • 9 years of code history
  • 3M quotes/year, 40% of market share, 4M of customers

Introduction

LesFurets service orchestration

LesFurets service orchestration

Insurers partners

We have 71 live insurers, on 5 products, each with business validation rules,
that filter prospects based on their profile

Insurer exclusions

Hierarchy of 492 legacy classes, no governance or auditability

Use case: insurer exclusions

Insurer exclusions based on object model (legacy code)

public void check(FieldContext context, FormuleMoto formule, Conducteur conducteur,
                  Vehicule vehicule, Void unused, Besoins besoins,
                  Set<EAbTestingScenario> scenarios)
                  throws ExclusionException {
    if (besoins == null) {
        return;
    }
    if (besoins.getDateDebutContrat() == null) {
        return;
    }
    if (!DateHelper.isAfter(besoins.getDateDebutContrat(),
                    DateHelper.ajouteJoursADate(DateHelper.getToday(), NBR_JOURS),
                    DateHelper.EPrecision.jour)) {
        throw new ExclusionException(DATE_EFFET_PLUS_60_JOURS);
    }
}

Use case: goal

  • compliance: the rules correspond to the specification documents
  • auditability: understand a rule without looking at the code
  • governance: maintenance of the rules catalogue
  • clarity: productivity for developers

Same rule, more fluent:

public ExclusionRule exclusionRule() {
    return DOOV.when(dateContrat().after(todayPlusDays(60)))
               .excludeFormules(F1, F2, F3)
               .withMessage(DATE_EFFET_PLUS_60_JOURS)
               .exclusionRule();
}

Why a fluent API?

Java is verbose, but you can reduce the noise and write code like natural language with a fluent API

// JUnit API
assertEquals(9, fellowshipOfTheRing.size());
assertTrue(fellowshipOfTheRing.contains(frodo, sam));
assertFalse(fellowshipOfTheRing.contains(sauron));

// AssertJ API (fluent)
assertThat(fellowshipOfTheRing).hasSize(9)
                               .contains(frodo, sam)
                               .doesNotContain(sauron);

Fluent API

New elements in Java 8 makes it easier to write a fluent API

// java.util.function (io.doov.core.dsl.impl.LogicalBinaryCondition)
left.predicate().and(right.predicate()).test(model, context)

// java.util.stream (io.doov.core.dsl.impl.LogicalNaryCondition)
steps.stream().anyMatch(s -> s.predicate().test(model, context))

// lambda and method reference (io.doov.core.dsl.impl.NumericCondition)
predicate(greaterThanMetadata(field, value),
          (model, context) -> Optional.ofNullable(value),
          (l, r) -> greaterThanFunction().apply(l, r));

Fluent API

Many popular libraries propose fluent APIs like
jOOQ, AssertJ, Apache Spark, etc.

Dataset<Row> averagePrice = prices
        .filter(value.<String>getAs("insurer")
                     .equals("COOL insurer"))
        .groupBy("product")
        .agg(avg("price").as("average"))
        .orderBy(desc("average"));

DSL

"A domain-specific language (DSL) is a computer language specialized to a particular application domain. This is in contrast to a general-purpose language (GPL), which is broadly applicable across domains"

Meet dOOv!

Domain Object Oriented Validation

dOOv is a fluent API for typesafe domain model validation

dOOv ecosystem

Model-map: deep accessors

We developed a typed, null-safe, two way mapping framework
from key value to model.

object

public class Account {

  @Path(field = SampleFieldId.LOGIN,
        readable = "account login")
  private String login;

  @Path(field = SampleFieldId.PASSWD,
        readable = "account password")
  private String password;

}

dictionary

Account account = new Account();
account.setLogin("alex");

SampleModel model = new SampleModel();
model.setAccount(account);

SampleModelWrapper wrapper 
    = new SampleModelWrapper(model);
wrapper.get(SampleFieldId.LOGIN); // alex
wrapper.set(SampleFieldId.LOGIN, "bob");
wrapper.get(SampleFieldId.LOGIN); // bob

How to write a validation rule?

key model

class Model {
  User user;
}

class User {
  @Path(field = EMAIL
        readable = ...)
  String email;
}

enum ModelFieldId {
  EMAIL;
}

code generate

class ModelFieldInfo {
  FieldInfo EMAIL;
}

class DslModel {
  StringFieldInfo userEmail;
}

class ModelWrapper {
  ...
}

write rules

class Validations {
  ValidationRule email = 
    DOOV.when(userEmail.eq(...))
        .validate()
        .register(REGISTRY_DEFAULT);
}

How to validate a model

wrap model

User user = new User();
user.setEmail("e@mail.com");

Model model = new Model();
model.setUser(user);

ModelWrapper wrap
  = new ModelWrapper(model);

execute

// Directly reference rule email
Validations.email.executeOn(wrap);

// Or use the registry
REGISTRY_DEFAULT.stream()
  .map(rule -> rule.executeOn(wrap));

Live code

public class Account extends Identity {

    @SamplePath(field = SampleFieldId.TIMEZONE,
                readable = "account timezone")
    private Timezone timezone;

    @SamplePath(field = SampleFieldId.PHONE_NUMBER,
                readable = "account phone number")
    private String phoneNumber;

    @SamplePath(field = SampleFieldId.EMAIL,
                readable = "account email")
    private String email;

    @SamplePath(field = SampleFieldId.EMAIL_ACCEPTED,
                readable = "account email accepted")
    private boolean acceptEmail;

    @SamplePath(field = SampleFieldId.EMAILS_PREFERENCES,
                readable = "account préférences mail")
    private Collection<EmailType> emailTypes = new HashSet<>();

}

Live code: rewrite with dOOv

public static boolean validateAccount(User user, Account account, Configuration config) {
    if (config == null) {
        return false;
    }
    if (user == null || user.getBirthDate() == null) {
        return false;
    }
    if (account == null || account.getCountry() == null || account.getPhoneNumber() == null) {
        return false;
    }
    if (YEARS.between(user.getBirthDate(), LocalDate.now()) >= 18
                    && account.getEmail().length() <= config.getMaxEmailSize()
                    && account.getCountry().equals(Country.FR) 
                    && account.getPhoneNumber().startsWith("+33")) {
        return true;
    }
    return false;
}

Live code: result

ValidationRule userAccount = DOOV
  // Entry point is when
  .when(userBirthdate.ageAt(today()).greaterOrEquals(18)
   .and(accountEmail.length().lesserOrEquals(configurationMaxEmailSize))
   .and(accountCountry.eq(Country.FR))
   .and(accountPhoneNumber.startsWith("+33")))
  // Terminal operation is validate
  .validate()
  // Optional: add to registry
  .registerOn(REGISTRY_DEFAULT);

Live code: test

@Test
public void should_user_account_validates() {
  // Condition assert
  assertThat(accountEmail().isNotNull()).validates(wrapper);

  // Rule assert
  assertThat(RulesConference.userAccount).validates(wrapper);

  // Result assert
  Result result = RulesConference.userAccount.executeOn(wrapper);
  assertThat(result).isTrue();
}

Live code: entry point

The entry point is DOOV#when(StepCondition) and the operation StepWhen#validate returns the validation rule

DOOV.when(accountEmail().matches("\\w+[@]\\w+\\.com")
      .or(accountEmail().matches("\\w+[@]\\w+\\.fr")))
    .validate()

This is lazy

Live code: natural language

A natural language version of the rule is available with ValidationRule#readable.

This makes auditability and compliance possible.

System.out.println(EMAIL_VALID.readable());
> When (email matches '\w+[@]\w+\.com' or email matches '\w+[@]\w+\.fr') validate

Live code: validation rule registry

You can add the rule in one or many registry with ValidationRule#registerOn(Registry)

This makes governance possible.

DOOV.when(accountEmail().matches("\\w+[@]\\w+\\.com")
      .or(accountEmail().matches("\\w+[@]\\w+\\.fr")))
   .validate()
   .registerOn(REGISTRY_ACCOUNT);

Live code: execution

The terminal operation ValidationRule#executeOn(FieldModel)
executes the rule

REGISTRY_ACCOUNT.stream()
    .map(rule -> rule.executeOn(model))
    .filter(Result::isInvalid)
    .map(Result::message)
    .collect(toList());

Live code: type safety

The available operations depend on the field type, and the arguments are type safe and validated by the compiler

DOOV.when(userAccountCreation().after(LocalDate.of(2000, 01, 01))).validate();
//                              ^^^^^
//                        only for date field
DOOV.when(userAccountCreation().after(LocalDate.of(2000, 01, 01))).validate();
//        ^^^^^^^^^^^^^^^^^^^         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
//             date field                  is type safe here

Syntax tree

Why a syntax tree?

Makes readable text generation possible:
we can output a multi-language rules catalog
in multiple formats (text, markdown, HTML, etc.)

LesFurets rules catalog

We generate a HTML validation rules catalog,
grouped by insurers and by insurance product

Rules catalog

The syntax tree makes it possible to generate the rule as text. Notice how the elements from the tree are tokenized (operator, fields, etc.)

Rules catalog

Also, the syntax tree makes it possible to see all the rules that applies for a specific field, for example the driver's date of birth.

Execution statistics

During execution, each node of the AST captures context value and predicate result. We know at runtime which node failed, and why.

Execution statistics

We make daily statistics that helps us shape the business,
by removing or tweaking rules as needed

Execution statistics

We also rewrite the execution rules by simplifying the predicate tree, to show the minimal predicate that fails. This is a complex problem we are currently working on.

Execution short-circuit

By default, predicate evaluation in dOOv behaves like Java: it short-circuits. This can be disabled to execute all nodes, even if they don't impact the end result.

Beyond bean validation

Performance - Java Microbenchmark Harness

We use JMH to check the performance of the DSL

Benchmark                          Mode  Samples      Mean  Mean error   Units
o.m.BenchmarkOldRule.valid_email  thrpt       25  1860.553      42.269  ops/ms
o.m.BenchmarkRule.valid_email     thrpt       25  1733.465      18.461  ops/ms

Performance of the DSL and POJO code are very close

Performance - comparing with other frameworks

We tested the performance of dOOv against the Bean Validation benchmark
and dOOv is faster in every category (hint: reflection API)

https://beastie.lesfurets.com/articles/doov-revisits-bean-validation-benchmark

Beyond Bean Validation: types

Bean Validation rules are not strongly typed since it's annotation based. This code will fail at compile time, but your IDE won't be able to tell you why.

public class Account {

    @NotNull @Email
    private Email email;

}

Beyond Bean Validation: complex validation

Because Bean Validation constraints are based on field annotation, cross validation between fields are only available through the extension mechanism.

public class Account {

    @Pattern(regexp = "(FR)|(UK)")
    private String country;

    @Pattern(regexp = "???")
    private String phoneNumber;

}

Beyond Bean Validation: natural language

Bean Validation rules are not written with a natural language syntax and does not provide a syntax tree

@Size(min = 10, max = 200,
      message = "About Me must be between 10 and 200 characters")
private String aboutMe;

DSL

userAboutMe().length().between(10, 200).validate().readable()
> When user about me length is between 10 and 200, validate

Beyond Bean Validation: syntax versus consistency

 

Bean Validation allows syntax validation
 
 
Bean Validation rules are written on the model
 
 
Bean Validation uses reflection
dOOv allows consistency checks
 
 
dOOv rules are writtent outside of the model, allowing complex checks
 
 
dOOv uses code generation

Conclusion

Goal

We migrated our 492 business rules to dOOv,
we now have compliance, auditability, governance, clarity
(and more!)

Roadmap

Next step is extending the DSL to create an object to object mapping framework, Domain Object Oriented Mapping (dOOm). It will feature the same AST to text and statistics functionnalities.

DOOV.map(userEmail().to(insurerEmail())
    .map(userFirstName(), userLastName())
      .using(StringJoiner)
      .to(insurerFullName())
    .when(userCountry().eq(FR))
      .map(userPhone().to(insurerPhone()))

Stay tuned for the next versions!

Enjoy dOOv.io

http://www.dOOv.io