DynamoDB QuerySpec {MaxResultSize + filter expression}











up vote
0
down vote

favorite












From the DynamoDB documentation




The Query operation allows you to limit the number of items that it
returns in the result. To do this, set the Limit parameter to the
maximum number of items that you want.



For example, suppose you Query a table, with a Limit value of 6, and
without a filter expression. The Query result will contain the first
six items from the table that match the key condition expression from
the request.



Now suppose you add a filter expression to the Query. In this case,
DynamoDB will apply the filter expression to the six items that were
returned, discarding those that do not match. The final Query result
will contain 6 items or fewer, depending on the number of items that
were filtered.




Looks like the following query should return (at least sometimes) 0 records.



In summary, I have a UserLogins table. A simplified version is:



1. UserId - HashKey
2. DeviceId - RangeKey
3. ActiveLogin - Boolean
4. TimeToLive - ...


Now, let's say UserId = X has 10,000 inactive logins in different DeviceIds and 1 active login.



However, when I run this query against my DynamoDB table:



QuerySpec{
hashKey: null,
rangeKeyCondition: null,
queryFilters: null,
nameMap: {"#0" -> "UserId"}, {"#1" -> "ActiveLogin"}
valueMap: {":0" -> "X"}, {":1" -> "true"}
exclusiveStartKey: null,
maxPageSize: null,
maxResultSize: 10,
req: {TableName: UserLogins,ConsistentRead: true,ReturnConsumedCapacity: TOTAL,FilterExpression: #1 = :1,KeyConditionExpression: #0 = :0,ExpressionAttributeNames: {#0=UserId, #1=ActiveLogin},ExpressionAttributeValues: {:0={S: X,}, :1={BOOL: true}}}


I always get 1 row. The 1 active login for UserId=X. And it's not happening just for 1 user, it's happening for multiple users in a similar situation.



Are my results contradicting the DynamoDB documentation?



It looks like a contradiction because if maxResultSize=10, means that DynamoDB will only read the first 10 items (out of 10,001) and then it will apply the filter active=true only (which might return 0 results). It seems very unlikely that the record with active=true happened to be in the first 10 records that DynamoDB read.



This is happening to hundreds of customers that are running similar queries. It works great, when according to the documentation it shouldn't be working.










share|improve this question




























    up vote
    0
    down vote

    favorite












    From the DynamoDB documentation




    The Query operation allows you to limit the number of items that it
    returns in the result. To do this, set the Limit parameter to the
    maximum number of items that you want.



    For example, suppose you Query a table, with a Limit value of 6, and
    without a filter expression. The Query result will contain the first
    six items from the table that match the key condition expression from
    the request.



    Now suppose you add a filter expression to the Query. In this case,
    DynamoDB will apply the filter expression to the six items that were
    returned, discarding those that do not match. The final Query result
    will contain 6 items or fewer, depending on the number of items that
    were filtered.




    Looks like the following query should return (at least sometimes) 0 records.



    In summary, I have a UserLogins table. A simplified version is:



    1. UserId - HashKey
    2. DeviceId - RangeKey
    3. ActiveLogin - Boolean
    4. TimeToLive - ...


    Now, let's say UserId = X has 10,000 inactive logins in different DeviceIds and 1 active login.



    However, when I run this query against my DynamoDB table:



    QuerySpec{
    hashKey: null,
    rangeKeyCondition: null,
    queryFilters: null,
    nameMap: {"#0" -> "UserId"}, {"#1" -> "ActiveLogin"}
    valueMap: {":0" -> "X"}, {":1" -> "true"}
    exclusiveStartKey: null,
    maxPageSize: null,
    maxResultSize: 10,
    req: {TableName: UserLogins,ConsistentRead: true,ReturnConsumedCapacity: TOTAL,FilterExpression: #1 = :1,KeyConditionExpression: #0 = :0,ExpressionAttributeNames: {#0=UserId, #1=ActiveLogin},ExpressionAttributeValues: {:0={S: X,}, :1={BOOL: true}}}


    I always get 1 row. The 1 active login for UserId=X. And it's not happening just for 1 user, it's happening for multiple users in a similar situation.



    Are my results contradicting the DynamoDB documentation?



    It looks like a contradiction because if maxResultSize=10, means that DynamoDB will only read the first 10 items (out of 10,001) and then it will apply the filter active=true only (which might return 0 results). It seems very unlikely that the record with active=true happened to be in the first 10 records that DynamoDB read.



    This is happening to hundreds of customers that are running similar queries. It works great, when according to the documentation it shouldn't be working.










    share|improve this question


























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      From the DynamoDB documentation




      The Query operation allows you to limit the number of items that it
      returns in the result. To do this, set the Limit parameter to the
      maximum number of items that you want.



      For example, suppose you Query a table, with a Limit value of 6, and
      without a filter expression. The Query result will contain the first
      six items from the table that match the key condition expression from
      the request.



      Now suppose you add a filter expression to the Query. In this case,
      DynamoDB will apply the filter expression to the six items that were
      returned, discarding those that do not match. The final Query result
      will contain 6 items or fewer, depending on the number of items that
      were filtered.




      Looks like the following query should return (at least sometimes) 0 records.



      In summary, I have a UserLogins table. A simplified version is:



      1. UserId - HashKey
      2. DeviceId - RangeKey
      3. ActiveLogin - Boolean
      4. TimeToLive - ...


      Now, let's say UserId = X has 10,000 inactive logins in different DeviceIds and 1 active login.



      However, when I run this query against my DynamoDB table:



      QuerySpec{
      hashKey: null,
      rangeKeyCondition: null,
      queryFilters: null,
      nameMap: {"#0" -> "UserId"}, {"#1" -> "ActiveLogin"}
      valueMap: {":0" -> "X"}, {":1" -> "true"}
      exclusiveStartKey: null,
      maxPageSize: null,
      maxResultSize: 10,
      req: {TableName: UserLogins,ConsistentRead: true,ReturnConsumedCapacity: TOTAL,FilterExpression: #1 = :1,KeyConditionExpression: #0 = :0,ExpressionAttributeNames: {#0=UserId, #1=ActiveLogin},ExpressionAttributeValues: {:0={S: X,}, :1={BOOL: true}}}


      I always get 1 row. The 1 active login for UserId=X. And it's not happening just for 1 user, it's happening for multiple users in a similar situation.



      Are my results contradicting the DynamoDB documentation?



      It looks like a contradiction because if maxResultSize=10, means that DynamoDB will only read the first 10 items (out of 10,001) and then it will apply the filter active=true only (which might return 0 results). It seems very unlikely that the record with active=true happened to be in the first 10 records that DynamoDB read.



      This is happening to hundreds of customers that are running similar queries. It works great, when according to the documentation it shouldn't be working.










      share|improve this question















      From the DynamoDB documentation




      The Query operation allows you to limit the number of items that it
      returns in the result. To do this, set the Limit parameter to the
      maximum number of items that you want.



      For example, suppose you Query a table, with a Limit value of 6, and
      without a filter expression. The Query result will contain the first
      six items from the table that match the key condition expression from
      the request.



      Now suppose you add a filter expression to the Query. In this case,
      DynamoDB will apply the filter expression to the six items that were
      returned, discarding those that do not match. The final Query result
      will contain 6 items or fewer, depending on the number of items that
      were filtered.




      Looks like the following query should return (at least sometimes) 0 records.



      In summary, I have a UserLogins table. A simplified version is:



      1. UserId - HashKey
      2. DeviceId - RangeKey
      3. ActiveLogin - Boolean
      4. TimeToLive - ...


      Now, let's say UserId = X has 10,000 inactive logins in different DeviceIds and 1 active login.



      However, when I run this query against my DynamoDB table:



      QuerySpec{
      hashKey: null,
      rangeKeyCondition: null,
      queryFilters: null,
      nameMap: {"#0" -> "UserId"}, {"#1" -> "ActiveLogin"}
      valueMap: {":0" -> "X"}, {":1" -> "true"}
      exclusiveStartKey: null,
      maxPageSize: null,
      maxResultSize: 10,
      req: {TableName: UserLogins,ConsistentRead: true,ReturnConsumedCapacity: TOTAL,FilterExpression: #1 = :1,KeyConditionExpression: #0 = :0,ExpressionAttributeNames: {#0=UserId, #1=ActiveLogin},ExpressionAttributeValues: {:0={S: X,}, :1={BOOL: true}}}


      I always get 1 row. The 1 active login for UserId=X. And it's not happening just for 1 user, it's happening for multiple users in a similar situation.



      Are my results contradicting the DynamoDB documentation?



      It looks like a contradiction because if maxResultSize=10, means that DynamoDB will only read the first 10 items (out of 10,001) and then it will apply the filter active=true only (which might return 0 results). It seems very unlikely that the record with active=true happened to be in the first 10 records that DynamoDB read.



      This is happening to hundreds of customers that are running similar queries. It works great, when according to the documentation it shouldn't be working.







      amazon-web-services amazon-dynamodb aws-sdk dynamodb-queries






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 5 at 2:15

























      asked Nov 5 at 1:59









      Cacho Santa

      5,41742760




      5,41742760
























          1 Answer
          1






          active

          oldest

          votes

















          up vote
          0
          down vote













          I can't see any obvious problem with the Query. Are you sure about your premise that users have 10,000 items each?



          Your keys are UserId and DeviceId. That seems to mean that if your user logs in with the same device it would overwrite the existing item. Or put another way, I think you are saying your users having 10,000 different devices each (unless the DeviceId rotates in some way).



          In your shoes I would just remove the filterexpression and print the results to the log to see what you're getting in your 10 results. Then remove the limit too and see what results you get with that.






          share|improve this answer





















            Your Answer






            StackExchange.ifUsing("editor", function () {
            StackExchange.using("externalEditor", function () {
            StackExchange.using("snippets", function () {
            StackExchange.snippets.init();
            });
            });
            }, "code-snippets");

            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "1"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            convertImagesToLinks: true,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: 10,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














             

            draft saved


            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147371%2fdynamodb-queryspec-maxresultsize-filter-expression%23new-answer', 'question_page');
            }
            );

            Post as a guest
































            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes








            up vote
            0
            down vote













            I can't see any obvious problem with the Query. Are you sure about your premise that users have 10,000 items each?



            Your keys are UserId and DeviceId. That seems to mean that if your user logs in with the same device it would overwrite the existing item. Or put another way, I think you are saying your users having 10,000 different devices each (unless the DeviceId rotates in some way).



            In your shoes I would just remove the filterexpression and print the results to the log to see what you're getting in your 10 results. Then remove the limit too and see what results you get with that.






            share|improve this answer

























              up vote
              0
              down vote













              I can't see any obvious problem with the Query. Are you sure about your premise that users have 10,000 items each?



              Your keys are UserId and DeviceId. That seems to mean that if your user logs in with the same device it would overwrite the existing item. Or put another way, I think you are saying your users having 10,000 different devices each (unless the DeviceId rotates in some way).



              In your shoes I would just remove the filterexpression and print the results to the log to see what you're getting in your 10 results. Then remove the limit too and see what results you get with that.






              share|improve this answer























                up vote
                0
                down vote










                up vote
                0
                down vote









                I can't see any obvious problem with the Query. Are you sure about your premise that users have 10,000 items each?



                Your keys are UserId and DeviceId. That seems to mean that if your user logs in with the same device it would overwrite the existing item. Or put another way, I think you are saying your users having 10,000 different devices each (unless the DeviceId rotates in some way).



                In your shoes I would just remove the filterexpression and print the results to the log to see what you're getting in your 10 results. Then remove the limit too and see what results you get with that.






                share|improve this answer












                I can't see any obvious problem with the Query. Are you sure about your premise that users have 10,000 items each?



                Your keys are UserId and DeviceId. That seems to mean that if your user logs in with the same device it would overwrite the existing item. Or put another way, I think you are saying your users having 10,000 different devices each (unless the DeviceId rotates in some way).



                In your shoes I would just remove the filterexpression and print the results to the log to see what you're getting in your 10 results. Then remove the limit too and see what results you get with that.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Nov 5 at 8:49









                Stu

                3,58131339




                3,58131339






























                     

                    draft saved


                    draft discarded



















































                     


                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147371%2fdynamodb-queryspec-maxresultsize-filter-expression%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest




















































































                    這個網誌中的熱門文章

                    Tangent Lines Diagram Along Smooth Curve

                    Yusuf al-Mu'taman ibn Hud

                    Zucchini