checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
---
2
2
SHA1:
3
- metadata.gz: 05fd12669243e14c2d6f1b1748acf461b8316dbb
4
- data.tar.gz: d1508fddee17b0a5c7eb0203b8dd1c9b52b74c66
3
+ metadata.gz: 58f0c90d4f03a5f3aa3d4ff363cdac165c948f8f
4
+ data.tar.gz: d927face94cf2c001534c1353b322294ab1984b8
5
5
SHA512:
6
- metadata.gz: 5ea75e62463c090f6806baedfd648768007c192d4e3a9e162c162083eb015cacb8eb6dfbc8ed0ce739bb9cc17c9a3889abc6b0bd02bb24f3f76470478996e85c
7
- data.tar.gz: 524dfe1e8c7ea123209746accfdb726ea0a6011f43c460cbcf2cff3c11531b2ee7afa99c26091d54ad8792dd05dca62aff2badaee5290babb1f72631ed0e94d0
6
+ metadata.gz: aa192b584a7ef8e58be570798c8d35ae04f17134791ed325a272f32dc5c32d3d847e60d06894da35093736d7c29f52bba32414b4d27b6d5107e2390e86c43221
7
+ data.tar.gz: cd329c1e3473f0f8a11f93bfc15bdd5278ebb041b8fccf6d5e2b7b496fa229919914b4080e4d97913e61ad090c6b67dfdf2a9c22e729cace4129e4f63e92bf95
data/README.md CHANGED
@@ -4,8 +4,9 @@
4
4
[![Code Climate](https://codeclimate.com/github/hsgubert/rails-sharding/badges/gpa.svg)](https://codeclimate.com/github/hsgubert/rails-sharding)
5
5
[![Test Coverage](https://codeclimate.com/github/hsgubert/rails-sharding/badges/coverage.svg)](https://codeclimate.com/github/hsgubert/rails-sharding/coverage)
6
6
[![Gem Version](https://badge.fury.io/rb/rails-sharding.svg)](https://badge.fury.io/rb/rails-sharding)
7
+ [![Dependency Status](https://gemnasium.com/badges/github.com/hsgubert/rails-sharding.svg)](https://gemnasium.com/github.com/hsgubert/rails-sharding)
7
8
8
- Simple and robust sharding for Rails, including Migrations and ActiveRecord extensions
9
+ Simple and robust sharding gem for Rails, including Migrations and ActiveRecord extensions
9
10
10
11
This gems allows you to easily create extra databases to your rails application, and freely allocate ActiveRecord instances to any of the databases. It also provides rake tasks and migrations to help you manage the schema by shard groups.
11
12
@@ -27,7 +28,10 @@ You can also use the block syntax, where all your queries inside will be directe
27
28
You can also pick and choose which models will be shardable, so that all the models that are not shardable will still be retrieved from the master database, even if inside a using_shard block.
28
29
29
30
## Compatibility
30
- As of now this gem has been tested only with Rails 4.2. It does not work yet with Rails 5.
31
+ Gem version 1.x.x -> compatible with Rails 5.0
32
+
33
+ Gem version 0.1.1 -> compatible with Rails 4.2
34
+
31
35
32
36
## Installation
33
37
@@ -83,7 +87,7 @@ rake shards:create
83
87
Go to the directory `db/shards_migrations/shard_group1` and add all migrations that you want to run on the shards of `shard_group1`. By design, all shards in a same group should always have the same schema. For example, add the following migration to your `db/shards_migrations/shard_group1`:
84
88
```ruby
85
89
# 20160808000000_create_users.rb
86
- class CreateClients < ActiveRecord::Migration
90
+ class CreateClients < ActiveRecord::Migration[5.0]
87
91
def up
88
92
create_table :users do |t|
89
93
t.string :username, :limit => 100
@@ -140,6 +144,29 @@ rake shards:migrate SHARD_GROUP=shard_group_1
140
144
rake shards:migrate SHARD_GROUP=shard_group_1 SHARD=shard1
141
145
```
142
146
147
+ ## Gem Options
148
+ Running the `rails g rails_sharding:scaffold` will create an initializer at `config/initializers/rails-sharding.rb`. You can pass additional configurations on this initializer to control the gem behavior. You can see below all available options and their default values:
149
+ ```ruby
150
+ # config/initializers/rails-sharding.rb
151
+
152
+ Rails::Sharding.setup do |config|
153
+ # If true one connection will be established per shard (in every shard group) on startup.
154
+ # If false the user must call Shards::ConnectionHandler.establish_connection(shard_group, shard_name) manually at least once before using each shard.
155
+ config.establish_all_connections_on_setup = true
156
+
157
+ # If true the method #using_shard will be mixed in ActiveRecord scopes. Put this to false if you don't want the gem to modify ActiveRecord
158
+ config.extend_active_record_scope = true,
159
+
160
+ # Specifies where to find the definition of the shards configurations
161
+ config.shards_config_file = 'config/shards.yml',
162
+
163
+ # Specifies where to find the migrations for each shard group
164
+ config.shards_migrations_dir = 'db/shards_migrations',
165
+
166
+ # Specifies where to find the schemas for each shard group
167
+ config.shards_schemas_dir = 'db/shards_schemas'
168
+ end
169
+ ```
143
170
144
171
## Development and Contributing
145
172
@@ -154,4 +181,4 @@ The gem is available as open source under the terms of the [MIT License](http://
154
181
155
182
## Acknowledgements
156
183
157
- This gem was inspired and based on several other gems like: [octopus](https://github.com/thiagopradi/octopus), [shard_handler](https://github.com/locaweb/shard_handler) and [active_record_shards](https://github.com/zendesk/active_record_shards).
184
+ This gem was inspired on several other gems like: [octopus](https://github.com/thiagopradi/octopus), [shard_handler](https://github.com/locaweb/shard_handler) and [active_record_shards](https://github.com/zendesk/active_record_shards).
data/Rakefile CHANGED
@@ -6,7 +6,8 @@ RSpec::Core::RakeTask.new(:spec)
6
6
task :default => :spec
7
7
8
8
# defines an environment task so we can run rake tasks from lib/tasks/rails-sharding.rake.
9
- # They require the :environment task, which is defined by rails, but we don need anything
9
+ # The tasks on rails-sharding.rake depend on the :environment task, which is usuallu defined
10
+ # by rails. In our case, we just stub it so the rake tasks run
10
11
task :environment do
11
12
# do nothing
12
13
end
@@ -16,12 +17,7 @@ namespace :db do
16
17
17
18
desc 'Loads gem test environment and rake tasks from gem'
18
19
task :load_env do
19
- require 'rspec'
20
+ require './spec/load_gem_test_env'
20
-
21
- # requires spec helper but ensures no test coverage is reported to codeclimate
22
- ENV['CODECLIMATE_REPO_TOKEN'] = nil
23
- require './spec/spec_helper'
24
-
25
21
load 'lib/tasks/rails-sharding.rake'
26
22
end
27
23
data/lib/generators/templates/shards.yml.example CHANGED
@@ -1,7 +1,7 @@
1
1
default: &default
2
2
adapter: mysql2
3
3
encoding: utf8
4
- reconnect: false
4
+ reconnect: true
5
5
pool: 5
6
6
username: ___
7
7
password: ___
data/lib/rails/sharding/config.rb CHANGED
@@ -13,6 +13,16 @@ module Rails::Sharding
13
13
# this to false if you don't want the gem to modify ActiveRecord
14
14
extend_active_record_scope: true,
15
15
16
+ # If true the query logs of ActiveRecord will be tagged with the corresponding
17
+ # shard you're querying
18
+ add_shard_tag_to_query_logs: true,
19
+
20
+ # If true a warning will be printed everytime a using_shard block ends without
21
+ # the shard connection being retrieved at least once inside the block. This warning
22
+ # is helpful to remember the developer to include ShardableModel module on the
23
+ # sharded models, otherwise they will always connect to the master database.
24
+ no_connection_retrieved_warning: true,
25
+
16
26
# Specifies where to find the definition of the shards configurations
17
27
shards_config_file: 'config/shards.yml',
18
28
@@ -27,6 +37,5 @@ module Rails::Sharding
27
37
self.cattr_accessor config_name
28
38
self.send(config_name.to_s + '=', default_value)
29
39
end
30
-
31
40
end
32
41
end
data/lib/rails/sharding/connection_handler.rb CHANGED
@@ -17,48 +17,64 @@ module Rails::Sharding
17
17
def self.establish_connection(shard_group, shard_name, environment=nil)
18
18
self.setup unless defined? @@connection_handler
19
19
20
- unless configurations = (environment.nil? ? Core.configurations : Core.configurations(environment))
20
+ configurations = (environment.nil? ? Core.configurations : Core.configurations(environment))
21
+ if configurations.nil?
21
22
raise Errors::ConfigNotFoundError, "Cannot find configuration for environment '#{environment}' in #{Config.shards_config_file}"
22
23
end
23
24
24
- unless shard_group_configurations = configurations[shard_group.to_s]
25
+ shard_group_configurations = configurations[shard_group.to_s]
26
+ if shard_group_configurations.nil?
25
27
raise Errors::ConfigNotFoundError, "Cannot find configuration for shard_group '#{shard_group}' in environment '#{environment}' in #{Config.shards_config_file}"
26
28
end
27
29
28
30
resolver = ActiveRecord::ConnectionAdapters::ConnectionSpecification::Resolver.new(shard_group_configurations)
29
31
begin
30
- connection_spec = resolver.spec(shard_name.to_sym)
31
- rescue ActiveRecord::AdapterNotSpecified => e
32
+ connection_name = connection_name(shard_group, shard_name)
33
+ connection_spec = resolver.spec(shard_name.to_sym, connection_name)
34
+ rescue ActiveRecord::AdapterNotSpecified
32
35
raise Errors::ConfigNotFoundError, "Cannot find configuration for shard '#{shard_group}:#{shard_name}' in environment '#{environment}' in #{Config.shards_config_file}"
33
36
end
34
37
35
- # since rails requires a class to be the connection owner, we trick rails passing
38
+ connection_handler.establish_connection(connection_spec)
36
- # an instance of the ConnectionPoolOwner class, that responds to the #name method
37
- connection_handler.establish_connection(connection_pool_owner(shard_group, shard_name), connection_spec)
38
39
end
39
40
40
41
def self.connection_pool(shard_group, shard_name)
41
- connection_handler.retrieve_connection_pool(connection_pool_owner(shard_group, shard_name))
42
- rescue Errors::ConnectionPoolRetrievalError
42
+ if connection_pool = connection_handler.retrieve_connection_pool(connection_name(shard_group, shard_name))
43
+ return connection_pool
44
+ end
45
+
43
46
# mimicking behavior of rails at:
44
- # https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb#507
45
- raise ActiveRecord::ConnectionNotEstablished, "No connection pool for shard #{connection_name(shard_group, shard_name)}"
47
+ # https://github.com/rails/rails/blob/v5.0.0.1/activerecord/lib/active_record/connection_handling.rb#124
48
+ raise ActiveRecord::ConnectionNotEstablished, "No connection pool for shard #{connection_name(shard_group, shard_name)}" if connection_pool.nil?
46
49
end
47
50
48
51
def self.retrieve_connection(shard_group, shard_name)
49
- connection_handler.retrieve_connection(connection_pool_owner(shard_group, shard_name))
52
+ connection_name = connection_name(shard_group, shard_name)
53
+ connection = connection_handler.retrieve_connection(connection_name)
54
+
55
+ if connection && Config.add_shard_tag_to_query_logs
56
+ add_shard_tag_to_connection_log(connection, connection_name)
57
+ else
58
+ connection
59
+ end
50
60
end
51
61
52
62
def self.connected?(shard_group, shard_name)
53
- connection_handler.connected?(connection_pool_owner(shard_group, shard_name))
63
+ connection_handler.connected?(connection_name(shard_group, shard_name))
54
64
end
55
65
56
66
def self.with_connection(shard_group, shard_name, &block)
57
- connection_pool(shard_group, shard_name).with_connection(&block)
67
+ connection_pool(shard_group, shard_name).with_connection do |connection|
68
+ if connection && Config.add_shard_tag_to_query_logs
69
+ connection_name = connection_name(shard_group, shard_name)
70
+ add_shard_tag_to_connection_log(connection, connection_name)
71
+ end
72
+ block.call(connection)
73
+ end
58
74
end
59
75
60
76
def self.remove_connection(shard_group, shard_name)
61
- connection_handler.remove_connection(connection_pool_owner(shard_group, shard_name))
77
+ connection_handler.remove_connection(connection_name(shard_group, shard_name))
62
78
end
63
79
64
80
private
@@ -70,12 +86,6 @@ module Rails::Sharding
70
86
71
87
def self.setup
72
88
@@connection_handler = ActiveRecord::ConnectionAdapters::ConnectionHandler.new
73
- @@connection_pool_owners = {}
74
- end
75
-
76
- def self.connection_pool_owner(shard_group, shard_name)
77
- connection_name = self.connection_name(shard_group, shard_name)
78
- @@connection_pool_owners[connection_name] ||= ConnectionPoolOwner.new(connection_name)
79
89
end
80
90
81
91
# Assembles connection name in the format "shard_group:shard_name"
@@ -83,22 +93,30 @@ module Rails::Sharding
83
93
shard_group.to_s + ':' + shard_name.to_s
84
94
end
85
95
86
- class ConnectionPoolOwner
87
- attr_reader :name
88
-
89
- def initialize(name)
90
- @name = name
96
+ # Adds a shard tag to the log of all queries executed through this connection
97
+ def self.add_shard_tag_to_connection_log(connection, shard_tag)
98
+ # avoids modifing connection twice
99
+ if connection.respond_to? :shard_tag
100
+ connection.shard_tag = shard_tag
101
+ return connection
91
102
end
92
103
93
- # Safeguard in case pool cannot be retrieved for owner. This makes the error clear
94
- def superclass
95
- raise Errors::ConnectionPoolRetrievalError, "ConnectionPool could not be retrieved for #{self}. See https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb#607"
104
+ # creates #shard_tag attribute in connection
105
+ connection.singleton_class.send(:attr_accessor, :shard_tag)
106
+ connection.shard_tag = shard_tag
96
- end
97
107
98
- # in case owner ends up printed by rails in an error message when retrieving connection
99
- def to_s
100
- "ConnectionPoolOwner with name #{self.name}"
108
+ # create an alias #original_execute, as a copy of the #execute for this connection
109
+ connection.singleton_class.send(:alias_method, :original_execute, :execute)
110
+
111
+ # defines a new #execute that adds a tag to the log
112
+ class << connection
113
+ def execute(sql, name=nil)
114
+ name = (name.to_s + " (#{shard_tag})").strip
115
+ self.original_execute(sql, name)
116
+ end
101
117
end
118
+
119
+ connection
102
120
end
103
121
end
104
122
end
data/lib/rails/sharding/core.rb CHANGED
@@ -10,23 +10,30 @@ module Rails::Sharding
10
10
11
11
# Opens a block where all queries will be directed to the selected shard
12
12
def self.using_shard(shard_group, shard_name)
13
- raise 'Cannot nest using_shard blocks' if ShardThreadRegistry.connecting_to_shard?
13
+ ShardThreadRegistry.push_current_shard(shard_group, shard_name)
14
-
15
- ShardThreadRegistry.current_shard_group = shard_group
16
- ShardThreadRegistry.current_shard_name = shard_name
17
14
yield
18
15
ensure
16
+ shard_group, shard_name, connection_used = ShardThreadRegistry.pop_current_shard
17
+
18
+ # shows warning to user
19
+ if Config.no_connection_retrieved_warning && !connection_used
20
+ puts "Warning: no connection to shard '#{shard_group}:#{shard_name}' was retrieved inside the using_shard block. Make sure you don't forget to include Rails::Sharding::ShardableModel to the models you want to be sharded. Disable this warning with Rails::Sharding::Config.no_connection_retrieved_warning = false."
21
+ end
22
+
19
23
# Releases connections in case user left some connection in the reserved state
20
24
# (by calling retrieve_connection instead of with_connection). Also, using
21
25
# normal activerecord queries leaves a connection in the reserved state
22
- ConnectionHandler.connection_pool(*ShardThreadRegistry.current_shard_group_and_name).release_connection
23
- ShardThreadRegistry.connect_back_to_master!
26
+ # Obs: don't do this with a master database connection
27
+ ConnectionHandler.connection_pool(shard_group, shard_name).release_connection if shard_group && shard_name
24
28
end
25
29
26
30
def self.configurations(environment=Rails.env)
27
31
@@db_configs ||= YAML.load_file(Config.shards_config_file)
28
- @@db_configs[environment]
29
- rescue Errno::ENOENT => e
32
+ environment_config = @@db_configs[environment]
33
+ return environment_config if environment_config
34
+
35
+ raise Errors::ConfigNotFoundError, 'Found no shard configurations for enviroment "' + environment + '" in ' + Config.shards_config_file.to_s + ' file was not found'
36
+ rescue Errno::ENOENT
30
37
raise Errors::ConfigNotFoundError, Config.shards_config_file.to_s + ' file was not found'
31
38
end
32
39
@@ -46,6 +53,23 @@ module Rails::Sharding
46
53
self.configurations[shard_group.to_s].keys
47
54
end
48
55
56
+ # yields a block for each shard in each shard group, with its configurations
57
+ # shard_group_filter: if passed yields only shards of this group
58
+ # shard_name_filter: if passed yields only shards with this name
59
+ def self.for_each_shard(shard_group_filter=nil, shard_name_filter=nil)
60
+ shard_group_filter.to_s if shard_group_filter
61
+ shard_name_filter.to_s if shard_name_filter
62
+
63
+ configurations.each do |shard_group, shards_configurations|
64
+ next if shard_group_filter && shard_group_filter != shard_group.to_s
65
+
66
+ shards_configurations.each do |shard, configuration|
67
+ next if shard_name_filter && shard_name_filter != shard.to_s
68
+ yield shard_group, shard, configuration
69
+ end
70
+ end
71
+ end
72
+
49
73
# Method that should be called on a rails initializer
50
74
def self.setup
51
75
if block_given?
data/lib/rails/sharding/shard_thread_registry.rb CHANGED
@@ -1,11 +1,21 @@
1
- require 'active_support/per_thread_registry'
2
1
3
2
module Rails::Sharding
4
3
class ShardThreadRegistry
5
- # creates two thread-specific variables
6
- extend ActiveSupport::PerThreadRegistry
7
- attr_accessor :_current_shard_group
8
- attr_accessor :_current_shard_name
4
+ # Creates two thread-specific stacks to store the shard of connection
5
+ # The top of the stack indicates the current connection
6
+ # This allows us to have nested blocks of #using_shard and keep track of the
7
+ # connections as we open/close those blocks
8
+ thread_mattr_accessor :_shard_group_stack
9
+ thread_mattr_accessor :_shard_name_stack
10
+
11
+ # auxiliary stack that keeps track of wether each shard connection was used
12
+ # inside its respective using_shard block (so we can print an alert if not)
13
+ thread_mattr_accessor :_shard_connection_used_stack
14
+
15
+ # accessors that initialize stacks if necessary
16
+ def self.shard_group_stack; self._shard_group_stack ||= [] end;
17
+ def self.shard_name_stack; self._shard_name_stack ||= [] end;
18
+ def self.shard_connection_used_stack; self._shard_connection_used_stack ||= [] end;
9
19
10
20
def self.connecting_to_master?
11
21
current_shard_group.nil? || current_shard_name.nil?
@@ -15,29 +25,48 @@ module Rails::Sharding
15
25
!connecting_to_master?
16
26
end
17
27
28
+ # Clears the connection stack and goes back to connecting to master
18
29
def self.connect_back_to_master!
19
- self.current_shard_group = nil
20
- self.current_shard_name = nil
30
+ shard_group_stack.clear
31
+ shard_name_stack.clear
32
+ shard_connection_used_stack.clear
21
33
end
22
34
23
35
# Returns the current shard group (for the current Thread)
24
36
def self.current_shard_group
25
- _current_shard_group
37
+ shard_group_stack.last
26
- end
27
-
28
- # Sets the current shard group (for the current Thread)
29
- def self.current_shard_group=(group)
30
- self._current_shard_group = group.blank? ? nil : group.to_sym
31
38
end
32
39
33
40
# Returns the current shard name (for the current Thread)
34
41
def self.current_shard_name
35
- _current_shard_name
42
+ shard_name_stack.last
43
+ end
44
+
45
+ def self.current_connection_used?
46
+ shard_connection_used_stack.last
47
+ end
48
+
49
+ # adds shard connection to the stack
50
+ def self.push_current_shard(group, name)
51
+ # this line supresses the unused connection warning when there are nested
52
+ # using_shard blocks. We suppress the warning because we view nested using_shard
53
+ # blocks as a override
54
+ notify_connection_retrieved
55
+
56
+ shard_group_stack.push(group.blank? ? nil : group.to_sym)
57
+ shard_name_stack.push(name.blank? ? nil : name.to_sym)
58
+ shard_connection_used_stack.push(false)
59
+ end
60
+
61
+ # notifies the current connection was used (wee keep track of this to warn
62
+ # the user in case the connection is not used)
63
+ def self.notify_connection_retrieved
64
+ shard_connection_used_stack[-1] = true if shard_connection_used_stack.present?
36
65
end
37
66
38
- # Sets the current shard name (for the current Thread)
39
- def self.current_shard_name=(name)
40
- self._current_shard_name = name.blank? ? nil : name.to_sym
67
+ # removes shard connection to the stack
68
+ def self.pop_current_shard
69
+ [shard_group_stack.pop, shard_name_stack.pop, shard_connection_used_stack.pop]
41
70
end
42
71
43
72
def self.current_shard_group_and_name
data/lib/rails/sharding/shardable_model.rb CHANGED
@@ -32,6 +32,8 @@ module Rails::Sharding
32
32
33
33
# @overrides ActiveRecord::ConnectionHandling#connection_pool
34
34
def sharded_connection_pool
35
+ ShardThreadRegistry.notify_connection_retrieved
36
+
35
37
if ShardThreadRegistry.connecting_to_master?
36
38
return original_connection_pool
37
39
else
@@ -41,6 +43,8 @@ module Rails::Sharding
41
43
42
44
# @overrides ActiveRecord::ConnectionHandling#retrieve_connection
43
45
def sharded_retrieve_connection
46
+ ShardThreadRegistry.notify_connection_retrieved
47
+
44
48
if ShardThreadRegistry.connecting_to_master?
45
49
return original_retrieve_connection
46
50
else
data/lib/rails/sharding/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
module Rails
2
2
module Sharding
3
- VERSION = "0.1.1"
3
+ VERSION = "1.0.1"
4
4
end
5
5
end
data/lib/tasks/rails-sharding.rake CHANGED
@@ -8,29 +8,36 @@ shards_namespace = namespace :shards do
8
8
ActiveRecord::Base.include(Rails::Sharding::ShardableModel) unless ActiveRecord::Base.ancestors.include? Rails::Sharding::ShardableModel
9
9
end
10
10
11
- desc "Creates database shards (options: RAILS_ENV=x SHARD_GROUP=x SHARD=x)"
12
- task create: [:environment] do
11
+ # for each of the shards, check that 1) the environment set in the ar_internal_metadata
12
+ # table matches the current rails env and 2) it is not a protected environment
13
+ # (defined in ActiveRecord::Base.protected_environments)
14
+ desc "Checks if the environment is not protected and if the shards match the current environment (options: RAILS_ENV=x SHARD_GROUP=x SHARD=x)"
15
+ task check_protected_environments: [:_make_activerecord_base_shardable] do
13
16
Rails::Sharding.configurations.each do |shard_group, shards_configurations|
14
17
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
15
18
16
- shards_configurations.each do |shard, configuration|
19
+ shards_configurations.each do |shard, _|
17
20
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
18
- puts "== Creating shard #{shard_group}:#{shard}"
19
- ActiveRecord::Tasks::DatabaseTasks.create(configuration)
21
+ Rails::Sharding.using_shard(shard_group, shard) do
22
+ ActiveRecord::Tasks::DatabaseTasks.check_protected_environments!
23
+ end
20
24
end
21
25
end
22
26
end
23
27
24
- desc "Drops database shards (options: RAILS_ENV=x SHARD_GROUP=x SHARD=x)"
25
- task drop: [:environment] do
26
- Rails::Sharding.configurations.each do |shard_group, shards_configurations|
27
- next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
28
+ desc "Creates database shards (options: RAILS_ENV=x SHARD_GROUP=x SHARD=x)"
29
+ task create: [:environment] do
30
+ Rails::Sharding.for_each_shard(ENV["SHARD_GROUP"], ENV["SHARD"]) do |shard_group, shard, configuration|
31
+ puts "== Creating shard #{shard_group}:#{shard}"
32
+ ActiveRecord::Tasks::DatabaseTasks.create(configuration)
33
+ end
34
+ end
28
35
29
- shards_configurations.each do |shard, configuration|
30
- next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
31
- puts "== Dropping shard #{shard_group}:#{shard}"
32
- ActiveRecord::Tasks::DatabaseTasks.drop(configuration)
33
- end
36
+ desc "Drops database shards (options: RAILS_ENV=x SHARD_GROUP=x SHARD=x)"
37
+ task drop: [:environment, :check_protected_environments] do
38
+ Rails::Sharding.for_each_shard(ENV["SHARD_GROUP"], ENV["SHARD"]) do |shard_group, shard, configuration|
39
+ puts "== Dropping shard #{shard_group}:#{shard}"
40
+ ActiveRecord::Tasks::DatabaseTasks.drop(configuration)
34
41
end
35
42
end
36
43
@@ -40,11 +47,9 @@ shards_namespace = namespace :shards do
40
47
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
41
48
42
49
# configures path for migrations of this shard group and creates dir if necessary
43
- shard_group_migrations_dir = File.join(Rails::Sharding::Config.shards_migrations_dir, shard_group.to_s)
50
+ setup_migrations_path(shard_group)
44
- ActiveRecord::Tasks::DatabaseTasks.migrations_paths = shard_group_migrations_dir
45
- FileUtils.mkdir_p(shard_group_migrations_dir)
46
51
47
- shards_configurations.each do |shard, configuration|
52
+ shards_configurations.each do |shard, _|
48
53
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
49
54
puts "== Migrating shard #{shard_group}:#{shard}"
50
55
Rails::Sharding.using_shard(shard_group, shard) do
@@ -78,18 +83,13 @@ shards_namespace = namespace :shards do
78
83
task dump: [:_make_activerecord_base_shardable] do
79
84
require "active_record/schema_dumper"
80
85
81
- Rails::Sharding.configurations.each do |shard_group, shards_configurations|
82
- next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
86
+ Rails::Sharding.for_each_shard(ENV["SHARD_GROUP"], ENV["SHARD"]) do |shard_group, shard, _configuration|
87
+ puts "== Dumping schema of #{shard_group}:#{shard}"
83
88
84
- shards_configurations.each do |shard, configuration|
85
- next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
86
- puts "== Dumping schema of #{shard_group}:#{shard}"
87
-
89
+ schema_filename = shard_schema_path(shard_group, shard)
90
+ File.open(schema_filename, "w:utf-8") do |file|
91
+ Rails::Sharding.using_shard(shard_group, shard) do
92
+ ActiveRecord::SchemaDumper.dump(Rails::Sharding::ConnectionHandler.retrieve_connection(shard_group, shard), file)
88
- schema_filename = shard_schema_path(shard_group, shard)
89
- File.open(schema_filename, "w:utf-8") do |file|
90
- Rails::Sharding.using_shard(shard_group, shard) do
91
- ActiveRecord::SchemaDumper.dump(Rails::Sharding::ConnectionHandler.retrieve_connection(shard_group, shard), file)
92
- end
93
93
end
94
94
end
95
95
end
@@ -100,13 +100,14 @@ shards_namespace = namespace :shards do
100
100
end
101
101
102
102
desc "Loads schema.rb file into the shards (options: RAILS_ENV=x, SHARD_GROUP=x, SHARD=x)"
103
- task load: [:_make_activerecord_base_shardable] do
103
+ task load: [:_make_activerecord_base_shardable, :check_protected_environments] do
104
104
Rails::Sharding.configurations.each do |shard_group, shards_configurations|
105
105
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
106
106
107
+ # configures path for migrations of this shard group and creates dir if necessary
107
108
setup_migrations_path(shard_group)
108
109
109
- shards_configurations.each do |shard, configuration|
110
+ shards_configurations.each do |shard, _|
110
111
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
111
112
puts "== Loading schema of #{shard_group}:#{shard}"
112
113
@@ -125,7 +126,7 @@ shards_namespace = namespace :shards do
125
126
end
126
127
127
128
namespace :migrate do
128
- desc 'Rollbacks the shards one migration and re migrate up (options: RAILS_ENV=x, VERSION=x, STEP=x, SHARD_GROUP=x, SHARD=x).'
129
+ desc 'Rollbacks the shards one migration and re migrate up (options: RAILS_ENV=x, VERSION=x, STEP=x, SHARD_GROUP=x, SHARD=x).'
129
130
task redo: [:environment] do
130
131
if ENV["VERSION"]
131
132
shards_namespace["migrate:down"].invoke
@@ -141,15 +142,15 @@ shards_namespace = namespace :shards do
141
142
142
143
desc 'Runs the "up" for a given migration VERSION.'
143
144
task up: [:_make_activerecord_base_shardable] do
144
- version = ENV["VERSION"] ? ENV["VERSION"].to_i : nil
145
+ version = get_version_or_else "VERSION is required"
145
- raise "VERSION is required" unless version
146
146
147
147
Rails::Sharding.configurations.each do |shard_group, shards_configurations|
148
148
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
149
149
150
+ # configures path for migrations of this shard group and creates dir if necessary
150
151
setup_migrations_path(shard_group)
151
152
152
- shards_configurations.each do |shard, configuration|
153
+ shards_configurations.each do |shard, _|
153
154
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
154
155
puts "== Migrating up shard #{shard_group}:#{shard}"
155
156
Rails::Sharding.using_shard(shard_group, shard) do
@@ -163,15 +164,15 @@ shards_namespace = namespace :shards do
163
164
164
165
desc 'Runs the "down" for a given migration VERSION.'
165
166
task down: [:_make_activerecord_base_shardable] do
166
- version = ENV["VERSION"] ? ENV["VERSION"].to_i : nil
167
+ version = get_version_or_else "VERSION is required - To go down one migration, run db:rollback"
167
- raise "VERSION is required - To go down one migration, run db:rollback" unless version
168
168
169
169
Rails::Sharding.configurations.each do |shard_group, shards_configurations|
170
170
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
171
171
172
+ # configures path for migrations of this shard group and creates dir if necessary
172
173
setup_migrations_path(shard_group)
173
174
174
- shards_configurations.each do |shard, configuration|
175
+ shards_configurations.each do |shard, _|
175
176
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
176
177
puts "== Migrating down shard #{shard_group}:#{shard}"
177
178
Rails::Sharding.using_shard(shard_group, shard) do
@@ -190,9 +191,10 @@ shards_namespace = namespace :shards do
190
191
Rails::Sharding.configurations.each do |shard_group, shards_configurations|
191
192
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
192
193
194
+ # configures path for migrations of this shard group and creates dir if necessary
193
195
setup_migrations_path(shard_group)
194
196
195
- shards_configurations.each do |shard, configuration|
197
+ shards_configurations.each do |shard, _|
196
198
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
197
199
puts "== Rolling back shard #{shard_group}:#{shard}"
198
200
Rails::Sharding.using_shard(shard_group, shard) do
@@ -206,15 +208,9 @@ shards_namespace = namespace :shards do
206
208
207
209
desc "Retrieves the current schema version number"
208
210
task version: [:_make_activerecord_base_shardable] do
209
- Rails::Sharding.configurations.each do |shard_group, shards_configurations|
210
- next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
211
-
211
+ Rails::Sharding.for_each_shard(ENV["SHARD_GROUP"], ENV["SHARD"]) do |shard_group, shard, _configuration|
212
+ Rails::Sharding.using_shard(shard_group, shard) do
213
+ puts "Shard #{shard_group}:#{shard} version: #{ActiveRecord::Migrator.current_version}"
212
- shards_configurations.each do |shard, configuration|
213
- next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
214
-
215
- Rails::Sharding.using_shard(shard_group, shard) do
216
- puts "Shard #{shard_group}:#{shard} version: #{ActiveRecord::Migrator.current_version}"
217
- end
218
214
end
219
215
end
220
216
end
@@ -226,9 +222,10 @@ shards_namespace = namespace :shards do
226
222
Rails::Sharding.test_configurations.each do |shard_group, shards_configurations|
227
223
next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
228
224
225
+ # configures path for migrations of this shard group and creates dir if necessary
229
226
setup_migrations_path(shard_group)
230
227
231
- shards_configurations.each do |shard, configuration|
228
+ shards_configurations.each do |shard, _|
232
229
next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
233
230
234
231
puts "== Loading test schema on shard #{shard_group}:#{shard}"
@@ -237,6 +234,10 @@ shards_namespace = namespace :shards do
237
234
should_reconnect = Rails::Sharding::ConnectionHandler.connection_pool(shard_group, shard).active_connection?
238
235
Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard, 'test')
239
236
237
+ # saves the current RAILS_ENV (we must change it so the environment is set correcly on the metadata table)
238
+ initial_rails_env = Rails.env
239
+ Rails.env = 'test'
240
+
240
241
schema_filename = shard_schema_path(shard_group, shard)
241
242
ActiveRecord::Tasks::DatabaseTasks.check_schema_file(schema_filename)
242
243
Rails::Sharding.using_shard(shard_group, shard) do
@@ -244,6 +245,9 @@ shards_namespace = namespace :shards do
244
245
load(schema_filename)
245
246
end
246
247
ensure
248
+ # restores rails env
249
+ Rails.env = initial_rails_env
250
+
247
251
if should_reconnect
248
252
# reestablishes connection for RAILS_ENV environment (whatever that is)
249
253
Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard)
@@ -262,26 +266,20 @@ shards_namespace = namespace :shards do
262
266
263
267
desc "Empty the test shards (drops all tables) (options: SHARD_GROUP=x, SHARD=x)"
264
268
task :purge => [:_make_activerecord_base_shardable] do
265
- Rails::Sharding.test_configurations.each do |shard_group, shards_configurations|
266
- next if ENV["SHARD_GROUP"] && ENV["SHARD_GROUP"] != shard_group.to_s
267
-
268
- shards_configurations.each do |shard, configuration|
269
- next if ENV["SHARD"] && ENV["SHARD"] != shard.to_s
270
-
269
+ Rails::Sharding.for_each_shard(ENV["SHARD_GROUP"], ENV["SHARD"]) do |shard_group, shard, configuration|
270
+ puts "== Purging test shard #{shard_group}:#{shard}"
271
+ begin
272
+ # establishes connection with test shard, saving if it was connected before (rails 4.2 doesn't do this, but should)
273
+ should_reconnect = Rails::Sharding::ConnectionHandler.connection_pool(shard_group, shard).active_connection?
274
+ Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard, 'test')
271
- puts "== Purging test shard #{shard_group}:#{shard}"
272
- begin
273
- # establishes connection with test shard, saving if it was connected before (rails 4.2 doesn't do this, but should)
274
- should_reconnect = Rails::Sharding::ConnectionHandler.connection_pool(shard_group, shard).active_connection?
275
- Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard, 'test')
276
275
277
- Rails::Sharding.using_shard(shard_group, shard) do
278
- ActiveRecord::Tasks::DatabaseTasks.purge(configuration)
279
- end
280
- ensure
281
- if should_reconnect
282
- # reestablishes connection for RAILS_ENV environment (whatever that is)
283
- Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard)
276
+ Rails::Sharding.using_shard(shard_group, shard) do
277
+ ActiveRecord::Tasks::DatabaseTasks.purge(configuration)
278
+ end
279
+ ensure
280
+ if should_reconnect
281
+ # reestablishes connection for RAILS_ENV environment (whatever that is)
282
+ Rails::Sharding::ConnectionHandler.establish_connection(shard_group, shard)
284
- end
285
283
end
286
284
end
287
285
end
@@ -304,4 +302,10 @@ shards_namespace = namespace :shards do
304
302
FileUtils.mkdir_p(shard_group_schemas_dir)
305
303
File.join(shard_group_schemas_dir, shard_name + "_schema.rb")
306
304
end
305
+
306
+ def get_version_or_else(error_message='VERSION is required')
307
+ version = ENV["VERSION"] ? ENV["VERSION"].to_i : nil
308
+ raise error_message unless version
309
+ version
310
+ end
307
311
end
data/rails-sharding.gemspec CHANGED
@@ -23,12 +23,12 @@ Gem::Specification.new do |spec|
23
23
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
24
24
spec.require_paths = ["lib"]
25
25
26
- spec.add_runtime_dependency 'rails', '>= 4.2', '< 5.0'
26
+ spec.add_runtime_dependency 'rails', '~> 5.0'
27
27
28
28
spec.add_development_dependency "bundler", "~> 1.12"
29
- spec.add_development_dependency "rake", "~> 10.0"
29
+ spec.add_development_dependency "rake", "~> 11.0"
30
30
spec.add_development_dependency "rspec", "~> 3.0"
31
- spec.add_development_dependency "byebug", '~> 0'
31
+ spec.add_development_dependency "byebug", '~> 9'
32
32
spec.add_development_dependency "mysql2", '~> 0'
33
33
spec.add_development_dependency "codeclimate-test-reporter", '~> 0'
34
34
end
metadata CHANGED
@@ -1,33 +1,27 @@
1
1
--- !ruby/object:Gem::Specification
2
2
name: rails-sharding
3
3
version: !ruby/object:Gem::Version
4
- version: 0.1.1
4
+ version: 1.0.1
5
5
platform: ruby
6
6
authors:
7
7
- Henrique Gubert
8
8
autorequire:
9
9
bindir: exe
10
10
cert_chain: []
11
- date: 2016-10-07 00:00:00.000000000 Z
11
+ date: 2016-11-02 00:00:00.000000000 Z
12
12
dependencies:
13
13
- !ruby/object:Gem::Dependency
14
14
name: rails
15
15
requirement: !ruby/object:Gem::Requirement
16
16
requirements:
17
- - - ">="
17
+ - - "~>"
18
- - !ruby/object:Gem::Version
19
- version: '4.2'
20
- - - "<"
21
18
- !ruby/object:Gem::Version
22
19
version: '5.0'
23
20
type: :runtime
24
21
prerelease: false
25
22
version_requirements: !ruby/object:Gem::Requirement
26
23
requirements:
27
- - - ">="
24
+ - - "~>"
28
- - !ruby/object:Gem::Version
29
- version: '4.2'
30
- - - "<"
31
25
- !ruby/object:Gem::Version
32
26
version: '5.0'
33
27
- !ruby/object:Gem::Dependency
@@ -50,14 +44,14 @@ dependencies:
50
44
requirements:
51
45
- - "~>"
52
46
- !ruby/object:Gem::Version
53
- version: '10.0'
47
+ version: '11.0'
54
48
type: :development
55
49
prerelease: false
56
50
version_requirements: !ruby/object:Gem::Requirement
57
51
requirements:
58
52
- - "~>"
59
53
- !ruby/object:Gem::Version
60
- version: '10.0'
54
+ version: '11.0'
61
55
- !ruby/object:Gem::Dependency
62
56
name: rspec
63
57
requirement: !ruby/object:Gem::Requirement
@@ -78,14 +72,14 @@ dependencies:
78
72
requirements:
79
73
- - "~>"
80
74
- !ruby/object:Gem::Version
81
- version: '0'
75
+ version: '9'
82
76
type: :development
83
77
prerelease: false
84
78
version_requirements: !ruby/object:Gem::Requirement
85
79
requirements:
86
80
- - "~>"
87
81
- !ruby/object:Gem::Version
88
- version: '0'
82
+ version: '9'
89
83
- !ruby/object:Gem::Dependency
90
84
name: mysql2
91
85
requirement: !ruby/object:Gem::Requirement