Let’s say you have a variable that you know is constant within each group, what is the best way to preserve it during a collapse operation in Stata? You might think taking the first value (firstnm) must be the fastest, since it theoretically only requires 1 step per group. If that is the case, you are in for a surprise—Stata is actually better in calculating the mean.<\/p>\n
Here are the simulation results for 100 groups of 1000 randomly generated observations, averaged over 30 runs:<\/p>\n
collapse mean 0.0443
\ncollapse median 0.1062
\ncollapse min 0.0844
\ncollapse max 0.0657
\ncollapse count 0.0456
\ncollapse firstnm 0.0473
\ncollapse lastnm 0.0464<\/p>\n
The measurements are reported in seconds. The relative speed is quite stable to variations in number of groups and observations. Base on my analysis of the underlying algorthims collapse uses, the reason why firstnm is so slow is that an order-preserving sort has to be performed on the data, and order-preserving sorts are slow relative to non-preserving ones. To confirm this is true, I ran the test with just one group of 100k observations:<\/p>\n
collapse mean 0.0614
\ncollapse firstnm 0.0508<\/p>\n
And as expected, firstnm is now faster. The calculation of mean also slows down more than that of firstnm as the number of groups decrease.<\/p>\n
Base on my simulations, calculation of mean is faster when there are as little as 3 groups, so mean is the way to go in most cases.<\/p>\n","protected":false},"excerpt":{"rendered":"
Let’s say you have a variable that you know is constant within each group, what is the best way to preserve it during a collapse operation in Stata? You might think taking the first value (firstnm) must be the fastest, since it theoretically only requires 1 step per group. If that is the case, you […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[2,7],"tags":[],"class_list":["post-1074","post","type-post","status-publish","format-standard","hentry","category-political-economy","category-tech-zone"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=\/wp\/v2\/posts\/1074","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1074"}],"version-history":[{"count":2,"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=\/wp\/v2\/posts\/1074\/revisions"}],"predecessor-version":[{"id":1076,"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=\/wp\/v2\/posts\/1074\/revisions\/1076"}],"wp:attachment":[{"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1074"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1074"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ticoneva.com\/journal\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1074"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}