Wsrtjtyk

I use apache ignite cache as data storage. Would like to know if there is a way to paginate a large data collection from the client. I do not need or want millions of records transfer from server to my web/mobile client.

private final ClientCache<UUID, Account> accounts;



public List<Account> getAll(int offset, int limit) 

{

    return accounts.query(new ScanQuery<UUID, Account>()

            .setLocal(false))

            .getAll()

            .stream()

            .skip(offset)

            .limit(limit)

            .map(entity -> entity.getValue())

            .collect(Collectors.toList());

}

Is this an efficient way?

I looked at using a Cursor but the API is limited to iterator...

Thanks.

I see a getAll() in your code. It makes all data be transferred to the caller side. This exactly what you wanted to avoid.

Iterator avoids this problem, because data is loaded in batches on demand. So, you don't have to load everything into memory of a single node, when you run the query. Page size may be configured by setting ScanQuery#pageSize property. By default it's equal to 1024. Iterator may be acquired by calling QueryCursor.iterator() method. So, instead of keeping the offset, you need to keep an iterator.

SQL SELECT query with LIMIT and OFFSET specified is also an option. But in case if you have more than one node, then LIMIT + OFFSET records will be loaded from each node to the reducer during the execution. You should take it into account.