feat: refactor metering ,support Third-party resource access and idempotent (#2258)

* feat:design metering、podresource、resource api type * metering watch user ns and create * feat:add e2e test about metering watch ns * feat:metering watch extensionResourcePrice * feat:pod-controller create resurce CR logic * feat:metering watch Resource create by resource-Controller and update Metering * feat:metering create AccountBalance and account will watch it and deduction balancde * feat:add infra calculate test * feat:add deploy.yaml and temporary ci * feat:change account ns to sealos-system * fix:update resourceName * feat:add deduction info to account chargelist * fix:payment error * fix:update resource name * Modify inappropriate places in the code by github Comment Signed-off-by: 晓杰 <2561589453@qq.com>
2026-09-01 15:38:06 +08:00 · 2022-12-29 14:43:06 +08:00
parent c0d082cce5
commit fd8ae0dfd6
72 changed files with 3970 additions and 374 deletions
@@ -13,5 +13,30 @@ resources:
  kind: Metering
  path: github.com/labring/sealos/controllers/metering/api/v1
  version: v1
+- api:
+    crdVersion: v1
+    namespaced: true
+  controller: true
+  domain: sealos.io
+  group: metering
+  kind: PodResource
+  path: github.com/labring/sealos/controllers/metering/api/v1
+  version: v1
+- api:
+    crdVersion: v1
+    namespaced: true
+  controller: true
+  domain: sealos.io
+  group: metering
+  kind: ExtensionResourcePrice
+  path: github.com/labring/sealos/controllers/metering/api/v1
+  version: v1
+- api:
+    crdVersion: v1
+    namespaced: true
+  domain: sealos.io
+  group: metering
+  kind: Resource
+  path: github.com/labring/sealos/controllers/metering/api/v1
+  version: v1
 version: "3"
-
@@ -1,94 +1,67 @@
-# metering
-// TODO(user): Add simple overview of use/purpose
+### 计费标准
+#### 查看阿里云按量计费

-## Description
-// TODO(user): An in-depth paragraph about your project and overview of use
-
-## Getting Started
-You’ll need a Kubernetes cluster to run against. You can use [KIND](https://sigs.k8s.io/kind) to get a local cluster for testing, or run against a remote cluster.
-**Note:** Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster `kubectl cluster-info` shows).
-
-### Running on the cluster
-1. Install Instances of Custom Resources:
-
-```sh
-kubectl apply -f config/samples/
+```
+计算型 c8y, 1vCPU 2GiB（ecs.c8y.small）0.133/小时
+通用型 g8y, 1vCPU 4GiB（ecs.g8y.small）0.200/小时
+计算型 c8y, 2vCPU 4GiB（ecs.c8y.large）0.267/小时
 ```

-2. Build and push your image to the location specified by `IMG`:
-	
-```sh
-make docker-build docker-push IMG=<some-registry>/metering:tag
-```
-	
-3. Deploy the controller to the cluster with the image specified by `IMG`:
+##### 磁盘按量计费标准

-```sh
-make deploy IMG=<some-registry>/metering:tag
+```
+ESSD云盘20GiBPL1  0.042/小时
 ```

-### Uninstall CRDs
-To delete the CRDs from the cluster:
+### 价格结论

-```sh
-make uninstall
-```
+**综合计算：cpu：0.067/单核/小时，内存：0.033/G/小时，磁盘 0.0021G/小时**

-### Undeploy controller
-UnDeploy the controller to the cluster:

-```sh
-make undeploy
-```

-## Contributing
-// TODO(user): Add detailed information on how you would like others to contribute to this project
+### 二、Metering 介绍

-### How it works
-This project aims to follow the Kubernetes [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
+## **一、背景**

-It uses [Controllers](https://kubernetes.io/docs/concepts/architecture/controller/) 
-which provides a reconcile function responsible for synchronizing resources untile the desired state is reached on the cluster 
+sealos cloud 是一个多租户的，以 k8s 为内核的云操作系统，每个用户都至少有一个自己的 namespace 用来使用，这样就给怎么计费带来挑战。怎么样计费 k8s 中 用户使用的 cpu、memory 等资源？怎么样计费流量等 Metering 不可见的资源。

-### Test It Out
-1. Install the CRDs into the cluster:
+## 二、需求

-```sh
-make install
-```
+计费正在使用的pod 的cpu、memory等资源，可以计费metering感知不到的第三方资源(如流量)

-2. Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):
+## 三、设计思路

-```sh
-make run
-```
+计量计费扣费解耦开，设计第三方资源计量计费接入方案

-**NOTE:** You can also run this in one step by running: `make install run`
+计量：计量使用的资源量

-### Modifying the API definitions
-If you are editing the API definitions, generate the manifests such as CRs or CRDs using:
+计费：根据资源价格和使用的资源量计算出价格

-```sh
-make manifests
-```
+扣费：从账户中扣除计算出的价格

-**NOTE:** Run `make --help` for more information on all potential `make` targets
+### 3.1、各模块介绍

-More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html)
+![](https://tva1.sinaimg.cn/large/008vxvgGly1h89ekci465j30l00b9757.jpg)

-## License
+**ExtensionResourcesPrice:**注册额外资源的声明，声明中有资源的单价和单位

-Copyright 2022.
+**meteringQuota:**计量模块，计量使用的资源量

-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
+**metering:计**费模块，每分钟去meteringQuota里面看使用的多少资源，根据价格计费。
+
+### 3.2、计量计费流程
+
+![](https://tva1.sinaimg.cn/large/008vxvgGly1h89elp5qjyj30pl0ehgnr.jpg)
+
+1、资源controller注册要进行计量的资源进ExtensionResourcePrice，ExtensionResourcePrice将要计量的资源注册进MeteringQuota，将计费的资源价格注册进Metering中。
+
+2、资源controller更新MeteringQuota中资源的used，增量资源把增加的部分更新进used，比如流量，这个时间段使用50M就used中增加50Mi，更新的时间间隔按照资源controller自己的需求自己更新，可以1个钟头更新一次，price那边价格定为每小时的价格就行。
+
+3、metering计费模块，每一分钟计费一次，查看对应的meteringQuota中资源的used，使用量/单价*价格
+
+就是最终价格，增量的存入totalAmout，并且把**meteringQuota里面的used置0**。
+
+4、deduction扣费模块可以按任意时间间隔扣费。

-    http://www.apache.org/licenses/LICENSE-2.0

-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.

@@ -0,0 +1,10 @@
+package v1
+
+const (
+	ExtensionResourcePricePrefix = "extensionresourceprice-"
+	MeteringPrefix               = "metering-"
+	ResourceQuotaPrefix          = "quota-"
+	METERINGNAMESPACEENV         = "METERING_SYSTEM_NAMESPACE"
+	DEFAULTMETERINGNAMESPACE     = "metering-system"
+	PodResourcePricePrefix       = "sealos-pod-controller"
+)
@@ -0,0 +1,61 @@
+/*
+Copyright 2022.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1
+
+import (
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// ExtensionResourcePriceSpec defines the desired state of ExtensionResourcePrice
+type ExtensionResourcePriceSpec struct {
+	ResourceName string                            `json:"resourceName,omitempty"`
+	Resources    map[v1.ResourceName]ResourcePrice `json:"resources,omitempty"`
+}
+
+// ExtensionResourcePriceStatus defines the observed state of ExtensionResourcePrice
+type ExtensionResourcePriceStatus struct {
+}
+
+//+kubebuilder:object:root=true
+//+kubebuilder:subresource:status
+
+// ExtensionResourcePrice is the Schema for the extensionresourceprices API
+type ExtensionResourcePrice struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   ExtensionResourcePriceSpec   `json:"spec,omitempty"`
+	Status ExtensionResourcePriceStatus `json:"status,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+
+// ExtensionResourcePriceList contains a list of ExtensionResourcePrice
+type ExtensionResourcePriceList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []ExtensionResourcePrice `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&ExtensionResourcePrice{}, &ExtensionResourcePriceList{})
+}
+
+func GetExtensionResourcePriceName(resourceControllerName string) string {
+	return ExtensionResourcePricePrefix + resourceControllerName
+}
@@ -17,10 +17,19 @@ limitations under the License.
 package v1

 import (
-	v1 "k8s.io/api/core/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )

+type TimeIntervalType string
+
+const (
+	MINUTE TimeIntervalType = "Minute"
+	HOUR   TimeIntervalType = "Hour"
+	DAY    TimeIntervalType = "Day"
+)
+
 /*
 apiVersion: v1
 kind: Metering
@@ -30,23 +39,45 @@ Spec:
    	owner: fanux
    	namespace: string
        resources: v1.ResourceList // resource type
+		timeInterval:59 // time interval
 */

 // MeteringSpec defines the desired state of Metering
-
 type MeteringSpec struct {
-	Namespace string          `json:"namespace"`
-	Owner     string          `json:"owner"`
-	Resources v1.ResourceList `json:"resources,omitempty"`
+	Namespace string `json:"namespace"`
+	Owner     string `json:"owner"`
+
+	//timeInterval unit is minutes
+	TimeInterval int                                          `json:"timeInterval,omitempty"`
+	Resources    map[corev1.ResourceName]ResourcePriceAndUsed `json:"resources,omitempty"`
 }

-type TimeIntervalType string
+// MeteringStatus defines the observed state of Metering
+type MeteringStatus struct {
+	BillingListH     []BillingList `json:"billingListH,omitempty"`
+	BillingListD     []BillingList `json:"billingListD,omitempty"`
+	TotalAmount      int64         `json:"totalAmount,omitempty"`
+	LatestUpdateTime int64         `json:"latestUpdateTime,omitempty"`
+	SeqID            int64         `json:"seqID,omitempty"`
+}

-const (
-	MINUTE TimeIntervalType = "Minute"
-	HOUR   TimeIntervalType = "Hour"
-	DAY    TimeIntervalType = "Day"
-)
+type ResourcePrice struct {
+	Unit     *resource.Quantity `json:"unit"`
+	Price    int64              `json:"price"` // 100 = 1¥
+	Describe string             `json:"describe,omitempty"`
+}
+
+type ResourcePriceAndUsed struct {
+	ResourcePrice `json:",inline"`
+	Used          *resource.Quantity `json:"used,omitempty"`
+}
+
+type ResourceMsg struct {
+	ResourceName corev1.ResourceName
+	Amount       float64
+	Used         *resource.Quantity
+	Unit         *resource.Quantity
+}

 type BillingList struct {
 	TimeStamp    int64            `json:"timeStamp,omitempty"`
@@ -55,18 +86,6 @@ type BillingList struct {
 	Amount       int64            `json:"amount,omitempty"`       //need to pay amount,100 = 1¥
 }

-// MeteringStatus defines the observed state of Metering
-
-type MeteringStatus struct {
-	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
-	// Important: Run "make" to regenerate code after modifying this file
-	BillingListM     []BillingList `json:"billingListM,omitempty"`
-	BillingListH     []BillingList `json:"billingListH,omitempty"`
-	BillingListD     []BillingList `json:"billingListD,omitempty"`
-	TotalAmount      int64         `json:"totalAmount,omitempty"`
-	LatestUpdateTime int64         `json:"latestUpdateTime,omitempty"`
-}
-
 //+kubebuilder:object:root=true
 //+kubebuilder:subresource:status
 // +kubebuilder:printcolumn:name="owner",type=string,JSONPath=".spec.owner"
@@ -94,3 +113,18 @@ type MeteringList struct {
 func init() {
 	SchemeBuilder.Register(&Metering{}, &MeteringList{})
 }
+
+func DefaultResourceQuota() corev1.ResourceList {
+	return corev1.ResourceList{
+		//corev1.ResourceRequestsCPU:    resource.MustParse("100"),
+		corev1.ResourceLimitsCPU: resource.MustParse("16"),
+		//corev1.ResourceRequestsMemory: resource.MustParse("100"),
+		corev1.ResourceLimitsMemory: resource.MustParse("64Gi"),
+		//For all PVCs, the total demand for storage resources cannot exceed this value
+		corev1.ResourceRequestsStorage: resource.MustParse("100Gi"),
+		//"limit.storage": resource.MustParse("100Gi"),
+		//Local ephemeral storage
+		corev1.ResourceLimitsEphemeralStorage: resource.MustParse("100Gi"),
+		//corev1.ResourceRequestsEphemeralStorage: resource.MustParse("100Gi"),
+	}
+}
@@ -0,0 +1,64 @@
+/*
+Copyright 2022.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1
+
+import (
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// PodResourceSpec defines the desired state of PodResource
+type PodResourceSpec struct {
+	ResourceName string                            `json:"resourceName,omitempty"`
+	Resources    map[v1.ResourceName]ResourcePrice `json:"resources,omitempty"`
+
+	// update used resources every Interval minutes
+	//+kubebuilder:default=50
+	//+kubebuilder:validation:Minimum=1
+	Interval int `json:"interval,omitempty"`
+}
+
+// PodResourceStatus defines the observed state of PodResource
+type PodResourceStatus struct {
+	LatestUpdateTime int64 `json:"latestUpdateTime,omitempty"`
+	SeqID            int64 `json:"seqID,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+//+kubebuilder:subresource:status
+
+// PodResource is the Schema for the podresources API
+type PodResource struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   PodResourceSpec   `json:"spec,omitempty"`
+	Status PodResourceStatus `json:"status,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+
+// PodResourceList contains a list of PodResource
+type PodResourceList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []PodResource `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&PodResource{}, &PodResourceList{})
+}
@@ -0,0 +1,71 @@
+/*
+Copyright 2022.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1
+
+import (
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+type Status string
+
+const (
+	Complete Status = "complete"
+	Create   Status = "create"
+)
+
+// ResourceSpec defines the desired state of Resource
+type ResourceSpec struct {
+	Resources map[v1.ResourceName]ResourceInfo `json:"resources,omitempty"`
+}
+
+type ResourceInfo struct {
+	Used      *resource.Quantity `json:"used,omitempty"`
+	TimeStamp int64              `json:"time,omitempty"`
+	NameSpace string             `json:"namespace,omitempty"`
+}
+
+// ResourceStatus defines the observed state of Resource
+type ResourceStatus struct {
+	Status Status `json:"status,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+//+kubebuilder:subresource:status
+
+// Resource is the Schema for the resources API
+type Resource struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   ResourceSpec   `json:"spec,omitempty"`
+	Status ResourceStatus `json:"status,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+
+// ResourceList contains a list of Resource
+type ResourceList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []Resource `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&Resource{}, &ResourceList{})
+}
@@ -41,6 +41,102 @@ func (in *BillingList) DeepCopy() *BillingList {
 	return out
 }

+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ExtensionResourcePrice) DeepCopyInto(out *ExtensionResourcePrice) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	out.Status = in.Status
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ExtensionResourcePrice.
+func (in *ExtensionResourcePrice) DeepCopy() *ExtensionResourcePrice {
+	if in == nil {
+		return nil
+	}
+	out := new(ExtensionResourcePrice)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ExtensionResourcePrice) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ExtensionResourcePriceList) DeepCopyInto(out *ExtensionResourcePriceList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]ExtensionResourcePrice, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ExtensionResourcePriceList.
+func (in *ExtensionResourcePriceList) DeepCopy() *ExtensionResourcePriceList {
+	if in == nil {
+		return nil
+	}
+	out := new(ExtensionResourcePriceList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ExtensionResourcePriceList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ExtensionResourcePriceSpec) DeepCopyInto(out *ExtensionResourcePriceSpec) {
+	*out = *in
+	if in.Resources != nil {
+		in, out := &in.Resources, &out.Resources
+		*out = make(map[corev1.ResourceName]ResourcePrice, len(*in))
+		for key, val := range *in {
+			(*out)[key] = *val.DeepCopy()
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ExtensionResourcePriceSpec.
+func (in *ExtensionResourcePriceSpec) DeepCopy() *ExtensionResourcePriceSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(ExtensionResourcePriceSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ExtensionResourcePriceStatus) DeepCopyInto(out *ExtensionResourcePriceStatus) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ExtensionResourcePriceStatus.
+func (in *ExtensionResourcePriceStatus) DeepCopy() *ExtensionResourcePriceStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(ExtensionResourcePriceStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Metering) DeepCopyInto(out *Metering) {
 	*out = *in
@@ -105,9 +201,9 @@ func (in *MeteringSpec) DeepCopyInto(out *MeteringSpec) {
 	*out = *in
 	if in.Resources != nil {
 		in, out := &in.Resources, &out.Resources
-		*out = make(corev1.ResourceList, len(*in))
+		*out = make(map[corev1.ResourceName]ResourcePriceAndUsed, len(*in))
 		for key, val := range *in {
-			(*out)[key] = val.DeepCopy()
+			(*out)[key] = *val.DeepCopy()
 		}
 	}
 }
@@ -125,11 +221,6 @@ func (in *MeteringSpec) DeepCopy() *MeteringSpec {
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *MeteringStatus) DeepCopyInto(out *MeteringStatus) {
 	*out = *in
-	if in.BillingListM != nil {
-		in, out := &in.BillingListM, &out.BillingListM
-		*out = make([]BillingList, len(*in))
-		copy(*out, *in)
-	}
 	if in.BillingListH != nil {
 		in, out := &in.BillingListH, &out.BillingListH
 		*out = make([]BillingList, len(*in))
@@ -151,3 +242,281 @@ func (in *MeteringStatus) DeepCopy() *MeteringStatus {
 	in.DeepCopyInto(out)
 	return out
 }
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *PodResource) DeepCopyInto(out *PodResource) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	out.Status = in.Status
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PodResource.
+func (in *PodResource) DeepCopy() *PodResource {
+	if in == nil {
+		return nil
+	}
+	out := new(PodResource)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *PodResource) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *PodResourceList) DeepCopyInto(out *PodResourceList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]PodResource, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PodResourceList.
+func (in *PodResourceList) DeepCopy() *PodResourceList {
+	if in == nil {
+		return nil
+	}
+	out := new(PodResourceList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *PodResourceList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *PodResourceSpec) DeepCopyInto(out *PodResourceSpec) {
+	*out = *in
+	if in.Resources != nil {
+		in, out := &in.Resources, &out.Resources
+		*out = make(map[corev1.ResourceName]ResourcePrice, len(*in))
+		for key, val := range *in {
+			(*out)[key] = *val.DeepCopy()
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PodResourceSpec.
+func (in *PodResourceSpec) DeepCopy() *PodResourceSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(PodResourceSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *PodResourceStatus) DeepCopyInto(out *PodResourceStatus) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PodResourceStatus.
+func (in *PodResourceStatus) DeepCopy() *PodResourceStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(PodResourceStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Resource) DeepCopyInto(out *Resource) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	out.Status = in.Status
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Resource.
+func (in *Resource) DeepCopy() *Resource {
+	if in == nil {
+		return nil
+	}
+	out := new(Resource)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *Resource) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceInfo) DeepCopyInto(out *ResourceInfo) {
+	*out = *in
+	if in.Used != nil {
+		in, out := &in.Used, &out.Used
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceInfo.
+func (in *ResourceInfo) DeepCopy() *ResourceInfo {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceInfo)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceList) DeepCopyInto(out *ResourceList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]Resource, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceList.
+func (in *ResourceList) DeepCopy() *ResourceList {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ResourceList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceMsg) DeepCopyInto(out *ResourceMsg) {
+	*out = *in
+	if in.Used != nil {
+		in, out := &in.Used, &out.Used
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+	if in.Unit != nil {
+		in, out := &in.Unit, &out.Unit
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceMsg.
+func (in *ResourceMsg) DeepCopy() *ResourceMsg {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceMsg)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourcePrice) DeepCopyInto(out *ResourcePrice) {
+	*out = *in
+	if in.Unit != nil {
+		in, out := &in.Unit, &out.Unit
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourcePrice.
+func (in *ResourcePrice) DeepCopy() *ResourcePrice {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourcePrice)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourcePriceAndUsed) DeepCopyInto(out *ResourcePriceAndUsed) {
+	*out = *in
+	in.ResourcePrice.DeepCopyInto(&out.ResourcePrice)
+	if in.Used != nil {
+		in, out := &in.Used, &out.Used
+		x := (*in).DeepCopy()
+		*out = &x
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourcePriceAndUsed.
+func (in *ResourcePriceAndUsed) DeepCopy() *ResourcePriceAndUsed {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourcePriceAndUsed)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceSpec) DeepCopyInto(out *ResourceSpec) {
+	*out = *in
+	if in.Resources != nil {
+		in, out := &in.Resources, &out.Resources
+		*out = make(map[corev1.ResourceName]ResourceInfo, len(*in))
+		for key, val := range *in {
+			(*out)[key] = *val.DeepCopy()
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceSpec.
+func (in *ResourceSpec) DeepCopy() *ResourceSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ResourceStatus) DeepCopyInto(out *ResourceStatus) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ResourceStatus.
+func (in *ResourceStatus) DeepCopy() *ResourceStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(ResourceStatus)
+	in.DeepCopyInto(out)
+	return out
+}
@@ -0,0 +1,75 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.8.0
+  creationTimestamp: null
+  name: extensionresourceprices.metering.sealos.io
+spec:
+  group: metering.sealos.io
+  names:
+    kind: ExtensionResourcePrice
+    listKind: ExtensionResourcePriceList
+    plural: extensionresourceprices
+    singular: extensionresourceprice
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: ExtensionResourcePrice is the Schema for the extensionresourceprices
+          API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ExtensionResourcePriceSpec defines the desired state of ExtensionResourcePrice
+            properties:
+              resourceName:
+                type: string
+              resources:
+                additionalProperties:
+                  properties:
+                    describe:
+                      type: string
+                    price:
+                      format: int64
+                      type: integer
+                    unit:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  required:
+                  - price
+                  - unit
+                  type: object
+                type: object
+            type: object
+          status:
+            description: ExtensionResourcePriceStatus defines the observed state of
+              ExtensionResourcePrice
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
@@ -44,6 +44,7 @@ spec:
          metadata:
            type: object
          spec:
+            description: MeteringSpec defines the desired state of Metering
            properties:
              namespace:
                type: string
@@ -51,18 +52,38 @@ spec:
                type: string
              resources:
                additionalProperties:
-                  anyOf:
-                  - type: integer
-                  - type: string
-                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                  x-kubernetes-int-or-string: true
-                description: ResourceList is a set of (resource name, quantity) pairs.
+                  properties:
+                    describe:
+                      type: string
+                    price:
+                      format: int64
+                      type: integer
+                    unit:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    used:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  required:
+                  - price
+                  - unit
+                  type: object
                type: object
+              timeInterval:
+                description: timeInterval unit is minutes
+                type: integer
            required:
            - namespace
            - owner
            type: object
          status:
+            description: MeteringStatus defines the observed state of Metering
            properties:
              billingListD:
                items:
@@ -94,27 +115,12 @@ spec:
                      type: integer
                  type: object
                type: array
-              billingListM:
-                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
-                  of cluster Important: Run "make" to regenerate code after modifying
-                  this file'
-                items:
-                  properties:
-                    amount:
-                      format: int64
-                      type: integer
-                    settled:
-                      type: boolean
-                    timeInterval:
-                      type: string
-                    timeStamp:
-                      format: int64
-                      type: integer
-                  type: object
-                type: array
              latestUpdateTime:
                format: int64
                type: integer
+              seqID:
+                format: int64
+                type: integer
              totalAmount:
                format: int64
                type: integer
@@ -0,0 +1,85 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.8.0
+  creationTimestamp: null
+  name: podresources.metering.sealos.io
+spec:
+  group: metering.sealos.io
+  names:
+    kind: PodResource
+    listKind: PodResourceList
+    plural: podresources
+    singular: podresource
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: PodResource is the Schema for the podresources API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: PodResourceSpec defines the desired state of PodResource
+            properties:
+              interval:
+                default: 50
+                description: update used resources every Interval minutes
+                minimum: 1
+                type: integer
+              resourceName:
+                type: string
+              resources:
+                additionalProperties:
+                  properties:
+                    describe:
+                      type: string
+                    price:
+                      format: int64
+                      type: integer
+                    unit:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  required:
+                  - price
+                  - unit
+                  type: object
+                type: object
+            type: object
+          status:
+            description: PodResourceStatus defines the observed state of PodResource
+            properties:
+              latestUpdateTime:
+                format: int64
+                type: integer
+              seqID:
+                format: int64
+                type: integer
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
@@ -0,0 +1,71 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.8.0
+  creationTimestamp: null
+  name: resources.metering.sealos.io
+spec:
+  group: metering.sealos.io
+  names:
+    kind: Resource
+    listKind: ResourceList
+    plural: resources
+    singular: resource
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: Resource is the Schema for the resources API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ResourceSpec defines the desired state of Resource
+            properties:
+              resources:
+                additionalProperties:
+                  properties:
+                    namespace:
+                      type: string
+                    time:
+                      format: int64
+                      type: integer
+                    used:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  type: object
+                type: object
+            type: object
+          status:
+            description: ResourceStatus defines the observed state of Resource
+            properties:
+              status:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
@@ -3,17 +3,26 @@
 # It should be run by config/default
 resources:
 - bases/metering.sealos.io_meterings.yaml
+- bases/metering.sealos.io_podresources.yaml
+- bases/metering.sealos.io_extensionresourceprices.yaml
+- bases/metering.sealos.io_resources.yaml
 #+kubebuilder:scaffold:crdkustomizeresource

 patchesStrategicMerge:
 # [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix.
 # patches here are for enabling the conversion webhook for each CRD
 #- patches/webhook_in_meterings.yaml
+#- patches/webhook_in_podresources.yaml
+#- patches/webhook_in_extensionresourceprices.yaml
+#- patches/webhook_in_resources.yaml
 #+kubebuilder:scaffold:crdkustomizewebhookpatch

 # [CERTMANAGER] To enable cert-manager, uncomment all the sections with [CERTMANAGER] prefix.
 # patches here are for enabling the CA injection for each CRD
 #- patches/cainjection_in_meterings.yaml
+#- patches/cainjection_in_podresources.yaml
+#- patches/cainjection_in_extensionresourceprices.yaml
+#- patches/cainjection_in_resources.yaml
 #+kubebuilder:scaffold:crdkustomizecainjectionpatch

 # the following config is for teaching kustomize how to do kustomization for CRDs.
@@ -0,0 +1,7 @@
+# The following patch adds a directive for certmanager to inject CA into the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: extensionresourceprices.metering.sealos.io
@@ -0,0 +1,7 @@
+# The following patch adds a directive for certmanager to inject CA into the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: podresources.metering.sealos.io
@@ -0,0 +1,7 @@
+# The following patch adds a directive for certmanager to inject CA into the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: resources.metering.sealos.io
@@ -0,0 +1,16 @@
+# The following patch enables a conversion webhook for the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: extensionresourceprices.metering.sealos.io
+spec:
+  conversion:
+    strategy: Webhook
+    webhook:
+      clientConfig:
+        service:
+          namespace: system
+          name: webhook-service
+          path: /convert
+      conversionReviewVersions:
+      - v1
@@ -0,0 +1,16 @@
+# The following patch enables a conversion webhook for the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: podresources.metering.sealos.io
+spec:
+  conversion:
+    strategy: Webhook
+    webhook:
+      clientConfig:
+        service:
+          namespace: system
+          name: webhook-service
+          path: /convert
+      conversionReviewVersions:
+      - v1
@@ -0,0 +1,16 @@
+# The following patch enables a conversion webhook for the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: resources.metering.sealos.io
+spec:
+  conversion:
+    strategy: Webhook
+    webhook:
+      clientConfig:
+        service:
+          namespace: system
+          name: webhook-service
+          path: /convert
+      conversionReviewVersions:
+      - v1
@@ -39,7 +39,7 @@ spec:
        - "--metrics-bind-address=127.0.0.1:8080"
        - "--leader-elect"
        env:
-          - name: SEALOS_SYSTEM_NAMESPACE
+          - name: METERING_SYSTEM_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
@@ -32,6 +32,7 @@ spec:
        args:
        - --leader-elect
        image: controller:latest
+        imagePullPolicy: Always
        name: manager
        securityContext:
          runAsNonRoot: true
@@ -0,0 +1,24 @@
+# permissions for end users to edit extensionresourceprices.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: extensionresourceprice-editor-role
+rules:
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - extensionresourceprices
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - extensionresourceprices/status
+  verbs:
+  - get
@@ -0,0 +1,20 @@
+# permissions for end users to view extensionresourceprices.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: extensionresourceprice-viewer-role
+rules:
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - extensionresourceprices
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - extensionresourceprices/status
+  verbs:
+  - get
@@ -0,0 +1,24 @@
+# permissions for end users to edit podresources.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: podresource-editor-role
+rules:
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources/status
+  verbs:
+  - get
@@ -0,0 +1,20 @@
+# permissions for end users to view podresources.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: podresource-viewer-role
+rules:
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources/status
+  verbs:
+  - get
@@ -0,0 +1,24 @@
+# permissions for end users to edit resources.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: resource-editor-role
+rules:
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - resources
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - resources/status
+  verbs:
+  - get
@@ -0,0 +1,20 @@
+# permissions for end users to view resources.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: resource-viewer-role
+rules:
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - resources
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - resources/status
+  verbs:
+  - get
@@ -5,6 +5,66 @@ metadata:
  creationTimestamp: null
  name: manager-role
 rules:
+- apiGroups:
+  - ""
+  resources:
+  - namespaces
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - pods
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - resourcequotas
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - infra.sealos.io
+  resources:
+  - infras
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - extensionresourceprices
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
 - apiGroups:
  - metering.sealos.io
  resources:
@@ -31,3 +91,53 @@ rules:
  - get
  - patch
  - update
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources/finalizers
+  verbs:
+  - update
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources/status
+  verbs:
+  - get
+  - patch
+  - update
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - resources
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
@@ -0,0 +1,11 @@
+apiVersion: metering.sealos.io/v1
+kind: ExtensionResourcePrice
+metadata:
+  name: extensionresourceprice-sample
+spec:
+  resourceName: pod
+  resources:
+    cpu:
+      unit: "10"
+      price: 67
+      describe: "cost per cpu per hour（price:100 = 1¥）"
@@ -0,0 +1,28 @@
+apiVersion: metering.sealos.io/v1
+kind: PodResource
+metadata:
+  name: podresourceprice-sample
+  namespace: metering-system
+spec:
+  resourceName: pod
+  interval: 60
+  resources:
+    cpu:
+      unit: "10"
+      price: 67
+      describe: "cost per cpu per hour（price:100 = 1¥）"
+
+    memory:
+      unit: "10G"
+      price: 33
+      describe: "the cost per gigabyte of memory per hour（price:100 = 1¥）"
+
+    storage:
+      unit: "100G"
+      price:  21
+      describe: "cost per gigabyte of storage per hour（price:100 = 1¥）"
+
+    ephemeral-storage:
+      unit: "100G"
+      price: 21
+      describe: "cost per gigabyte of storage per hour（price:100 = 1¥）"
@@ -0,0 +1,9 @@
+apiVersion: metering.sealos.io/v1
+kind: Resource
+metadata:
+  name: resource-sample
+spec:
+  resources:
+    resourceName: "cpu"
+    Used: 1
+
@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: test3
+  annotations:
+    user.sealos.io/creator: 0ffc5a5d-12c3-4e3c-a71f-58abbb50047c
@@ -20,12 +20,17 @@ import (
 	"context"
 	"fmt"
 	"os"
+	"strconv"
 	"time"

+	"k8s.io/client-go/util/retry"
+
+	"k8s.io/apimachinery/pkg/api/resource"
+	"k8s.io/apimachinery/pkg/types"
+
 	"github.com/go-logr/logr"
 	infrav1 "github.com/labring/sealos/controllers/infra/api/v1"
 	userv1 "github.com/labring/sealos/controllers/user/api/v1"
-	"k8s.io/apimachinery/pkg/api/resource"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
 	"sigs.k8s.io/controller-runtime/pkg/source"
@@ -33,25 +38,11 @@ import (
 	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"

 	corev1 "k8s.io/api/core/v1"
-	"k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"

 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
-
-	"sigs.k8s.io/controller-runtime/pkg/log"
-)
-
-var SealosSystemNamespace string
-
-func init() {
-	SealosSystemNamespace = os.Getenv("SEALOS_SYSTEM_NAMESPACE")
-}
-
-const (
-	FinalizerName          = "metering.sealos.io/finalizer"
-	UserAnnotationOwnerKey = "user.sealos.io/creator"
 )

 // MeteringReconcile reconciles a Metering object
@@ -59,171 +50,209 @@ type MeteringReconcile struct {
 	client.Client
 	Scheme *runtime.Scheme
 	logr.Logger
+	MeteringSystemNameSpace string
+	MeteringInterval        int
 }

 //+kubebuilder:rbac:groups=metering.sealos.io,resources=meterings,verbs=get;list;watch;create;update;patch;delete
 //+kubebuilder:rbac:groups=metering.sealos.io,resources=meterings/status,verbs=get;update;patch
 //+kubebuilder:rbac:groups=metering.sealos.io,resources=meterings/finalizers,verbs=update
+//+kubebuilder:rbac:groups=metering.sealos.io,resources=extensionresourceprices,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=metering.sealos.io,resources=resources,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=core,resources=namespaces,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=core,resources=resourcequotas,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=infra.sealos.io,resources=infras,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=user.sealos.io,resources=accountbalances,verbs=get;list;watch;create;update;patch;delete

-// Reconcile is part of the main kubernetes reconciliation loop which aims to
-// move the current state of the cluster closer to the desired state.
-// the Metering object against the actual cluster state, and then
-// perform operations to make the cluster state reflect the state specified by
-// the user.
-//
-// For more details, check Reconcile and its Result here:
-// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
-
-func DefaultResourceQuota() corev1.ResourceList {
-	return corev1.ResourceList{
-		corev1.ResourceCPU:          resource.MustParse("1"),
-		corev1.ResourceMemory:       resource.MustParse("1Gi"),
-		corev1.ResourceLimitsCPU:    resource.MustParse("1"),
-		corev1.ResourceLimitsMemory: resource.MustParse("1Gi"),
-	}
-}
-
-// metering belong to account，resourceQuota belong to metering
-
+// Reconcile metering belong to account，resourceQuota belong to metering
 func (r *MeteringReconcile) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
-	r.Logger = log.FromContext(ctx)
-	var metering meteringv1.Metering
 	r.Logger.Info("enter reconcile", "name: ", req.Name, "namespace: ", req.Namespace)
+	// when user ns create enter this logic
+	var ns corev1.Namespace
+	if err := r.Get(ctx, req.NamespacedName, &ns); err == nil {
+		// if create a user namespace ,will enter this reconcile
+		if _, ok := ns.Annotations[userv1.UserAnnotationOwnerKey]; ok {
+			if ns.DeletionTimestamp != nil {
+				r.Logger.Info("namespace is deleting,want to delete metering", "namespace", ns.Name)
+				err := r.DelMetering(ctx, meteringv1.MeteringPrefix+ns.Name, r.MeteringSystemNameSpace)
+				return ctrl.Result{}, err
+			}
+			if err := r.initMetering(ctx, ns); err != nil {
+				return ctrl.Result{}, err
+			}
+		}
+		return ctrl.Result{}, nil
+	} else if client.IgnoreNotFound(err) != nil {
+		return ctrl.Result{}, err
+	}

-	// if create a user namespace ,will enter this reconcile
-	if req.Name != SealosSystemNamespace {
-		var ns corev1.Namespace
-		err := r.Get(ctx, req.NamespacedName, &ns)
+	// when extensionResourcePrice change will enter this logic
+	var extensionResourcePrice meteringv1.ExtensionResourcePrice
+	if err := r.Get(ctx, req.NamespacedName, &extensionResourcePrice); err == nil {
+		meteringList := &meteringv1.MeteringList{}
+		err = r.List(ctx, meteringList)
 		if err != nil {
 			return ctrl.Result{}, client.IgnoreNotFound(err)
 		}
-		if _, ok := ns.Annotations[UserAnnotationOwnerKey]; !ok {
-			r.Logger.Info(fmt.Sprintf("not found owner of namespace name: %v", ns.Name))
+		for _, metering := range meteringList.Items {
+			if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &metering, func() error {
+				if metering.Spec.Resources == nil {
+					metering.Spec.Resources = make(map[corev1.ResourceName]meteringv1.ResourcePriceAndUsed, 0)
+				}
+				for resourceName, v := range extensionResourcePrice.Spec.Resources {
+					metering.Spec.Resources[resourceName] = meteringv1.ResourcePriceAndUsed{ResourcePrice: v, Used: &resource.Quantity{}}
+				}
+				//r.Logger.Info("metering", "metering", metering)
+				return nil
+			}); err != nil {
+				return ctrl.Result{}, fmt.Errorf("sync metering failed: %v", err)
+			}
+		}
+		//r.Logger.Info("", "meteringList", meteringList)
+	} else if client.IgnoreNotFound(err) != nil {
+		return ctrl.Result{}, err
+	}
+
+	// when Resource create will enter this logic
+	resources := &meteringv1.Resource{}
+	if err := r.Get(ctx, req.NamespacedName, resources); err == nil {
+		if resources.Status.Status == meteringv1.Complete || !resources.DeletionTimestamp.IsZero() {
 			return ctrl.Result{}, nil
 		}
-		owner := ns.Annotations[UserAnnotationOwnerKey]
-		if metering, err = r.createMetering(ctx, owner, ns); err != nil {
-			r.Logger.Error(err, "Failed to get Metering")
-			return ctrl.Result{}, client.IgnoreNotFound(err)
-		}
+		r.Logger.Info("enter update resource used", "resource name: ", req.Name, "resource namespace: ", req.Namespace)
+		var metering meteringv1.Metering
+		for resourceName, resourceInfo := range resources.Spec.Resources {
+			if err := r.Get(ctx, types.NamespacedName{Namespace: r.MeteringSystemNameSpace, Name: meteringv1.MeteringPrefix + resourceInfo.NameSpace}, &metering); err != nil {
+				r.Logger.Error(err, "get metering failed", "name", meteringv1.MeteringPrefix+resourceInfo.NameSpace)
+				continue
+			}
+			if _, ok := metering.Spec.Resources[resourceName]; !ok {
+				r.Logger.Error(err, "resource not found in metering", "name", resourceName)
+				continue
+			}
+			if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &metering, func() error {
+				metering.Spec.Resources[resourceName].Used.Add(*resourceInfo.Used)
+				return r.Update(ctx, &metering)
+			}); err != nil {
+				return ctrl.Result{Requeue: true}, err
+			}

-		// get or create resourceQuota
-		if err = r.syncResourceQuota(ctx, metering); err != nil {
+			resources.Status.Status = meteringv1.Complete
+			err := r.Update(ctx, resources)
+			if err != nil {
+				return ctrl.Result{Requeue: true}, err
+			}
+		}
+	} else if client.IgnoreNotFound(err) != nil {
+		return ctrl.Result{}, err
+	}
+
+	var metering meteringv1.Metering
+	if err := r.Get(ctx, req.NamespacedName, &metering); err != nil {
+		return ctrl.Result{}, client.IgnoreNotFound(err)
+	}
+
+	if time.Now().Unix()-metering.Status.LatestUpdateTime >= int64(time.Minute.Seconds())*int64(metering.Spec.TimeInterval) {
+		//r.Logger.Info("enter update metering", "metering name:", req.Name, "metering namespace:", req.Namespace, "lastUpdate Time", metering.Status.LatestUpdateTime, "now", time.Now().Unix(), "diff", time.Now().Unix()-metering.Status.LatestUpdateTime, "interval", int64(time.Minute.Seconds())*int64(metering.Spec.TimeInterval))
+		totalAccount, err := r.CalculateCost(ctx, &metering)
+		if err != nil {
+			r.Logger.Error(err, err.Error())
 			return ctrl.Result{}, err
 		}
-	} else {
-		if err := r.Get(ctx, req.NamespacedName, &metering); err != nil {
-			return ctrl.Result{}, client.IgnoreNotFound(err)
+
+		if err := r.clearResourceUsed(ctx, &metering); err != nil {
+			return ctrl.Result{}, err
+		}
+
+		if err := r.syncAccountBalance(ctx, metering.Spec.Owner, totalAccount, metering.Status.SeqID); err != nil {
+			r.Logger.Error(err, err.Error())
+			return ctrl.Result{}, fmt.Errorf("meteringName:%v,amount:%v,err:%v", metering.Name, totalAccount, err)
+		}
+		if err := r.updateBillingList(ctx, totalAccount, &metering); err != nil {
+			r.Logger.Error(err, err.Error())
+			return ctrl.Result{}, err
 		}
-		//// metering finalizer
-		//if metering.DeletionTimestamp.IsZero() {
-		//	if controllerutil.AddFinalizer(&metering, FinalizerName) == true {
-		//		//TODO delete resource quota
-		//		if err := r.Update(ctx, &metering); err != nil {
-		//			return ctrl.Result{}, err
-		//		}
-		//	}
-		//} else {
-		//	if controllerutil.RemoveFinalizer(&metering, FinalizerName) == true {
-		//		if err := r.Update(ctx, &metering); err != nil {
-		//			return ctrl.Result{}, err
-		//		}
-		//	}
-		//	return ctrl.Result{}, nil
-		//}
-	}
-	// get or create resourceQuota
-	if err := r.syncResourceQuota(ctx, metering); err != nil {
-		return ctrl.Result{}, err
 	}

-	r.Logger.Info("update BillingList")
-	err := r.updateBillingList(ctx, &metering)
+	return ctrl.Result{Requeue: true, RequeueAfter: time.Minute * time.Duration(r.MeteringInterval)}, nil
+}
+
+func (r *MeteringReconcile) CalculateCost(ctx context.Context, metering *meteringv1.Metering) (int64, error) {
+	var resourceMsgs []meteringv1.ResourceMsg
+	infraAmount, err := r.calculateInfraCost(ctx, metering)
 	if err != nil {
-		r.Logger.Error(err, err.Error())
-		return ctrl.Result{}, err
+		r.Logger.Error(err, "Failed to get InfraCost")
+	} else {
+		resourceMsgs = append(resourceMsgs, meteringv1.ResourceMsg{
+			ResourceName: "infra",
+			Amount:       float64(infraAmount),
+		})
 	}
-	// Ensure the deployment size is the same as the spec
-	return ctrl.Result{Requeue: true, RequeueAfter: time.Minute}, nil
+
+	var totalAmount float64
+	for resourceName, resourceValue := range metering.Spec.Resources {
+		if _, ok := metering.Spec.Resources[resourceName]; ok {
+			amount := float64(resourceValue.Used.MilliValue()) * float64(resourceValue.Price) / float64(resourceValue.Unit.MilliValue())
+			resourceMsgs = append(resourceMsgs, meteringv1.ResourceMsg{
+				ResourceName: resourceName,
+				Amount:       amount,
+				Used:         resourceValue.Used,
+				Unit:         resourceValue.Unit,
+			})
+			totalAmount += amount
+		} else {
+			r.Logger.Error(fmt.Errorf("resource: %v is not found", resourceName), "Failed to get Resource")
+		}
+	}
+	totalAmount += float64(infraAmount)
+	r.Logger.Info(fmt.Sprintf("meteringNmae %v,resourceMsg: %+v", metering.Name, resourceMsgs), "totalAmount", totalAmount)
+	return int64(totalAmount), nil
 }

-func (r *MeteringReconcile) createMetering(ctx context.Context, owner string, ns corev1.Namespace) (meteringv1.Metering, error) {
-	var metering meteringv1.Metering
-	err := r.Get(ctx, client.ObjectKey{Namespace: SealosSystemNamespace, Name: fmt.Sprintf("metering-%v", ns.Name)}, &metering)
-	if errors.IsNotFound(err) {
-		r.Logger.Info("creat metering")
-		metering = meteringv1.Metering{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:      fmt.Sprintf("metering-%v", ns.Name),
-				Namespace: SealosSystemNamespace,
-			},
-			Spec: meteringv1.MeteringSpec{
-				Namespace: ns.Name,
-				Owner:     owner,
-				Resources: DefaultResourceQuota(),
-			},
-			Status: meteringv1.MeteringStatus{
-				TotalAmount:      0,
-				LatestUpdateTime: time.Now().Unix(),
-			},
-		}
-		if err := r.Create(ctx, &metering); err != nil {
-			return meteringv1.Metering{}, fmt.Errorf("create metering failed: %v", err)
-		}
-	} else if err != nil {
-		r.Logger.Error(err, "Failed to get Metering")
-		return meteringv1.Metering{}, client.IgnoreNotFound(err)
+func (r *MeteringReconcile) calculateInfraCost(ctx context.Context, metering *meteringv1.Metering) (int64, error) {
+	infraList := &infrav1.InfraList{}
+	err := r.List(ctx, infraList)
+	//r.Logger.Info("infraList", "infraList", infraList)
+	if client.IgnoreNotFound(err) != nil {
+		r.Logger.Error(err, "Failed to get InfraList")
+		return 0, err
 	}
-	return metering, nil
+	var amount int64
+	for _, infra := range infraList.Items {
+		if infra.Namespace == metering.Spec.Namespace {
+			count, err := infra.QueryPrice()
+			r.Logger.Info("get infra :", "name: ", infra.Name, "namespace: ", infra.Namespace, "price: ", count, "meteringName", metering.Name, "userNamespace", metering.Spec.Namespace)
+			if err != nil {
+				r.Logger.Error(err, "Failed to get InfraPrice", "meteringName", metering.Name)
+			}
+			amount += count
+		}
+	}
+	// amount is preHour,but billing is perMinute
+	r.Logger.V(1).Info("infra", "price:", amount)
+	return amount, nil
 }

-func (r *MeteringReconcile) syncResourceQuota(ctx context.Context, metering meteringv1.Metering) error {
-	quota := &corev1.ResourceQuota{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      metering.Name,
-			Namespace: metering.Spec.Namespace,
-		},
-	}
-	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, quota, func() error {
-		quota.Spec.Hard = metering.Spec.Resources
+func (r *MeteringReconcile) clearResourceUsed(ctx context.Context, metering *meteringv1.Metering) error {
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, metering, func() error {
+		for _, v := range metering.Spec.Resources {
+			used := resource.MustParse(v.Used.String())
+			v.Used.Sub(used)
+		}
 		return nil
 	}); err != nil {
-		return fmt.Errorf("sync resource quota failed: %v", err)
+		return err
 	}
 	return nil
 }

-func (r *MeteringReconcile) updateBillingList(ctx context.Context, metering *meteringv1.Metering) error {
-	// check timeInterval is after 1 minute
-	if time.Now().Unix()-metering.Status.LatestUpdateTime >= time.Minute.Milliseconds()/1000 {
-		infraList := &infrav1.InfraList{}
-		err := r.List(ctx, infraList)
-		if err != nil {
+func (r *MeteringReconcile) updateBillingList(ctx context.Context, amount int64, metering *meteringv1.Metering) error {
+	//r.Logger.Info("enter metering updateBillingList", "metering name: ", metering.Name, "metering namespace: ", metering.Namespace, "lastupdat Time", metering.Status.LatestUpdateTime)
+	if err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		if err := r.Get(ctx, types.NamespacedName{Name: metering.Name, Namespace: metering.Namespace}, metering); err != nil {
 			return err
 		}
-		var amount int64
-		for _, infra := range infraList.Items {
-			count, err := infra.QueryPrice()
-			r.Logger.Info("get infra :", "name: ", infra.Name, "namespace: ", infra.Namespace, "price: ", count)
-			if err != nil {
-				return err
-			}
-			amount += count
-		}
-		// amount is preHour,but billing is perMinute
-		amount /= 60
-		// TODO Billing for resources such as memory and hard disks under namesapce
-
-		metering.Status.BillingListM = append(metering.Status.BillingListM, NewBillingList(meteringv1.MINUTE, amount))
-		if len(metering.Status.BillingListM) >= 60 {
-			var totalAmountH int64
-			for i := 0; i < 60; i++ {
-				totalAmountH += metering.Status.BillingListM[i].Amount
-			}
-			metering.Status.BillingListM = metering.Status.BillingListM[60:]
-			metering.Status.BillingListH = append(metering.Status.BillingListH, NewBillingList(meteringv1.HOUR, totalAmountH))
-		}
-
+		metering.Status.BillingListH = append(metering.Status.BillingListH, NewBillingList(meteringv1.HOUR, amount))
 		if len(metering.Status.BillingListH) >= 24 {
 			var totalAmountD int64
 			for i := 0; i < 24; i++ {
@@ -232,37 +261,13 @@ func (r *MeteringReconcile) updateBillingList(ctx context.Context, metering *met
 			metering.Status.BillingListH = metering.Status.BillingListH[24:]
 			metering.Status.BillingListD = append(metering.Status.BillingListD, NewBillingList(meteringv1.DAY, totalAmountD))
 		}
-		metering.Status.LatestUpdateTime = time.Now().Unix()
 		metering.Status.TotalAmount += amount
-		err = r.Status().Update(ctx, metering)
-		if err != nil {
-			return err
-		}
-
-		account := &userv1.Account{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:      metering.Spec.Owner,
-				Namespace: SealosSystemNamespace,
-			},
-		}
-		err = r.Get(ctx, client.ObjectKey{Namespace: SealosSystemNamespace, Name: account.Name}, account)
-		if err != nil {
-			r.Logger.Error(err, "get account err", "account", account)
-			return err
-		}
-		//deduct balance
-		account.Status.Balance -= amount
-
-		err = r.Status().Update(ctx, account)
-		if err != nil {
-			return err
-		}
-		r.Logger.Info("DebitSuccess ,", "user:", account.Name, "balance", account.Status.Balance)
-		//TODO reportNoMoneyError
-		//if account.Status.Balance < 0 {
-		//}
+		metering.Status.LatestUpdateTime = time.Now().Unix()
+		metering.Status.SeqID++
+		return r.Status().Update(ctx, metering)
+	}); err != nil {
+		return err
 	}
-
 	return nil
 }

@@ -275,10 +280,151 @@ func NewBillingList(TimeInterval meteringv1.TimeIntervalType, amount int64) mete
 	}
 }

+func (r *MeteringReconcile) initMetering(ctx context.Context, ns corev1.Namespace) error {
+	totalResourcePrice, err := r.GetAllExtensionResources(ctx)
+	if err != nil {
+		return err
+	}
+	r.Logger.Info("totalResourcePrice", "totalResourcePrice", totalResourcePrice)
+	if err = r.syncMetering(ctx, ns, totalResourcePrice); err != nil {
+		r.Logger.Error(err, "Failed to create Metering")
+		return client.IgnoreNotFound(err)
+	}
+	return nil
+}
+
+func (r *MeteringReconcile) syncMetering(ctx context.Context, ns corev1.Namespace, resourcePrice map[corev1.ResourceName]meteringv1.ResourcePriceAndUsed) error {
+	metering := meteringv1.Metering{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      meteringv1.MeteringPrefix + ns.Name,
+			Namespace: r.MeteringSystemNameSpace,
+		},
+	}
+
+	r.Logger.V(1).Info("metering env", "METERING_INTERVAL", r.MeteringSystemNameSpace, "timeInterval: ", r.MeteringInterval)
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &metering, func() error {
+		metering.Spec.Namespace = ns.Name
+		metering.Spec.Owner = ns.Annotations[userv1.UserAnnotationOwnerKey]
+		metering.Spec.Resources = resourcePrice
+		metering.Spec.TimeInterval = r.MeteringInterval
+		return nil
+	}); err != nil {
+		r.Error(err, "Failed to update Metering")
+		return err
+	}
+	r.Logger.Info("metering Spec", "metering.Spec", metering.Spec)
+	// get or create resourceQuota
+	if err := r.syncResourceQuota(ctx, metering); err != nil {
+		r.Error(err, "Failed to syncResourceQuota")
+		return err
+	}
+	return nil
+}
+
+func (r *MeteringReconcile) syncResourceQuota(ctx context.Context, metering meteringv1.Metering) error {
+	quota := &corev1.ResourceQuota{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      meteringv1.ResourceQuotaPrefix + metering.Spec.Namespace,
+			Namespace: metering.Spec.Namespace,
+		},
+	}
+
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, quota, func() error {
+		quota.Spec.Hard = meteringv1.DefaultResourceQuota()
+		return nil
+	}); err != nil {
+		return fmt.Errorf("sync resource quota failed: %v", err)
+	}
+	return nil
+}
+
+func (r *MeteringReconcile) DelMetering(ctx context.Context, name, namespace string) error {
+	metering := meteringv1.Metering{}
+	err := r.Get(ctx, types.NamespacedName{Name: name, Namespace: namespace}, &metering)
+	if err != nil {
+		return client.IgnoreNotFound(err)
+	}
+	err = r.Delete(ctx, &metering)
+	if err != nil {
+		return client.IgnoreNotFound(err)
+	}
+	return nil
+}
+
+func (r *MeteringReconcile) GetAllExtensionResources(ctx context.Context) (map[corev1.ResourceName]meteringv1.ResourcePriceAndUsed, error) {
+	extensionResourcesPriceList := meteringv1.ExtensionResourcePriceList{}
+	totalResourcePrice := make(map[corev1.ResourceName]meteringv1.ResourcePriceAndUsed, 0)
+	err := r.List(ctx, &extensionResourcesPriceList)
+	if err != nil {
+		return totalResourcePrice, client.IgnoreNotFound(err)
+	}
+	for _, extensionResourcesPrice := range extensionResourcesPriceList.Items {
+		for k, v := range extensionResourcesPrice.Spec.Resources {
+			totalResourcePrice[k] = meteringv1.ResourcePriceAndUsed{ResourcePrice: v, Used: &resource.Quantity{}}
+		}
+	}
+	return totalResourcePrice, nil
+}
+
+func (r *MeteringReconcile) syncAccountBalance(ctx context.Context, owner string, amount int64, seqID int64) error {
+	if amount == 0 {
+		return nil
+	} else if amount < 0 {
+		return fmt.Errorf("deduction amount is <0")
+	}
+
+	accountBalance := userv1.AccountBalance{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      fmt.Sprintf("%s-%s-%v", userv1.AccountBalancePrefix, owner, seqID),
+			Namespace: r.MeteringSystemNameSpace,
+		},
+	}
+
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &accountBalance, func() error {
+		accountBalance.Spec.Owner = owner
+		accountBalance.Spec.TimeStamp = time.Now().Unix()
+		accountBalance.Spec.Amount = amount
+		accountBalance.Status.Status = meteringv1.Create
+		return nil
+	}); err != nil {
+		return err
+	}
+	return nil
+}
+
 // SetupWithManager sets up the controller with the Manager.
 func (r *MeteringReconcile) SetupWithManager(mgr ctrl.Manager) error {
+	const controllerName = "metering-controller"
+	r.Logger = ctrl.Log.WithName(controllerName)
+	r.Logger.V(1).Info("reconcile controller metering")
+
+	// get env METERING_SYSTEM_NAMESPACE and METERING_INTERVAL
+	r.MeteringSystemNameSpace = os.Getenv(meteringv1.METERINGNAMESPACEENV)
+	if os.Getenv(meteringv1.METERINGNAMESPACEENV) == "" {
+		r.MeteringSystemNameSpace = meteringv1.DEFAULTMETERINGNAMESPACE
+	}
+
+	timeInterval := os.Getenv("METERING_INTERVAL")
+
+	if timeInterval == "" || timeInterval == "0" {
+		timeInterval = DefaultInterval()
+	}
+	r.MeteringInterval, _ = strconv.Atoi(timeInterval)
+	if r.MeteringInterval == 0 {
+		r.MeteringInterval = DefaultIntervalInt()
+	}
+	r.Logger.Info("metering env", meteringv1.METERINGNAMESPACEENV, r.MeteringSystemNameSpace, "timeinterval:", r.MeteringInterval, "envtimeinterval", os.Getenv("METERING_INTERVAL"))
 	return ctrl.NewControllerManagedBy(mgr).
 		For(&meteringv1.Metering{}).
 		Watches(&source.Kind{Type: &corev1.Namespace{}}, &handler.EnqueueRequestForObject{}).
+		Watches(&source.Kind{Type: &meteringv1.ExtensionResourcePrice{}}, &handler.EnqueueRequestForObject{}).
+		Watches(&source.Kind{Type: &meteringv1.Resource{}}, &handler.EnqueueRequestForObject{}).
 		Complete(r)
 }
+
+func DefaultInterval() string {
+	return "60"
+}
+func DefaultIntervalInt() int {
+	return 60
+}
@@ -0,0 +1,233 @@
+/*
+Copyright 2022.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package controllers
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/go-logr/logr"
+	userv1 "github.com/labring/sealos/controllers/user/api/v1"
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/util/retry"
+	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
+
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+)
+
+// PodResourceReconciler reconciles a PodResource object
+type PodResourceReconciler struct {
+	client.Client
+	Scheme *runtime.Scheme
+	logr.Logger
+	MeteringSystemNameSpace string
+}
+
+//+kubebuilder:rbac:groups=metering.sealos.io,resources=podresources,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=metering.sealos.io,resources=podresources/status,verbs=get;update;patch
+//+kubebuilder:rbac:groups=metering.sealos.io,resources=podresources/finalizers,verbs=update
+//+kubebuilder:rbac:groups=metering.sealos.io,resources=resources,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=core,resources=namespaces,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch;create;update;patch;delete
+
+func (r *PodResourceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+	podController := &meteringv1.PodResource{}
+	err := r.Get(ctx, req.NamespacedName, podController)
+	if err != nil {
+		return ctrl.Result{}, client.IgnoreNotFound(err)
+	}
+
+	err = r.CreateOrUpdateExtensionResourcesPrice(ctx, podController)
+	if err != nil {
+		r.Logger.Error(err, "CreateOrUpdateExtensionResourcesPrice failed")
+		return ctrl.Result{Requeue: true, RequeueAfter: time.Second}, client.IgnoreNotFound(err)
+	}
+	// update resource used every podController.Spec.Interval Minutes
+	if time.Now().Unix()-podController.Status.LatestUpdateTime >= int64(time.Minute.Minutes())*int64(podController.Spec.Interval) {
+		if err := r.UpdateResourceUsed(ctx, podController); err != nil {
+			r.Logger.Error(err, "UpdateResourceUsed failed")
+			return ctrl.Result{Requeue: true}, client.IgnoreNotFound(err)
+		}
+	}
+	return ctrl.Result{Requeue: true, RequeueAfter: time.Duration(podController.Spec.Interval) * time.Minute}, nil
+}
+
+// CreateOrUpdateExtensionResourcesPrice need to create a ExtensionResourcesPrice to make metering-quota know this resource
+func (r *PodResourceReconciler) CreateOrUpdateExtensionResourcesPrice(ctx context.Context, obj client.Object) error {
+	podController := obj.(*meteringv1.PodResource)
+	extensionResourcesPrice := &meteringv1.ExtensionResourcePrice{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: r.MeteringSystemNameSpace,
+			Name:      meteringv1.ExtensionResourcePricePrefix + meteringv1.PodResourcePricePrefix,
+		},
+	}
+
+	r.Logger.Info("create or update extensionResourcePrice", "podController name", podController.Name, "podController Resources", podController.Spec.Resources)
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, extensionResourcesPrice, func() error {
+		extensionResourcesPrice.Spec.Resources = podController.Spec.Resources
+		extensionResourcesPrice.Spec.ResourceName = podController.Spec.ResourceName
+		//extensionResourcesPrice.SetPrice(apiVersion, kind, podController.Name)
+		return controllerutil.SetControllerReference(podController, extensionResourcesPrice, r.Scheme)
+	}); err != nil {
+		return fmt.Errorf("sync ExtensionResourcesPrice failed: %v", err)
+	}
+	//r.Logger.V(1).Info("sync extensionResourcesPrice", "extensionResourcesPrice.Spec", extensionResourcesPrice.Spec)
+	return nil
+}
+
+func (r *PodResourceReconciler) UpdateResourceUsed(ctx context.Context, obj client.Object) error {
+	podController := obj.(*meteringv1.PodResource)
+	var podList v1.PodList
+	err := r.List(ctx, &podList)
+	if err != nil {
+		return err
+	}
+
+	for _, pod := range podList.Items {
+		if !r.checkPodStatus(pod) || !r.checkPodNamespace(pod) {
+			continue
+		}
+		for _, con := range pod.Spec.Containers {
+			for resourceName := range podController.Spec.Resources {
+				var resourceQuantity resource.Quantity
+				var ok bool
+				if resourceQuantity, ok = r.checkResourceExist(resourceName, con); !ok {
+					continue
+				}
+				if err := r.syncResource(ctx, pod, con.Name, resourceName, &resourceQuantity, *podController); err != nil {
+					r.Logger.Error(err, "syncResource failed")
+				}
+			}
+		}
+
+		// storage resource not in pod container, so need to get it from resource quota
+		resourceQuota := v1.ResourceQuota{}
+		if err := r.Get(ctx, client.ObjectKey{Name: meteringv1.ResourceQuotaPrefix + pod.Namespace, Namespace: pod.Namespace}, &resourceQuota); err != nil {
+			return err
+		}
+		//r.Logger.V(1).Info("resourceQuota", "resourceQuota", resourceQuota)
+		storage := resourceQuota.Status.Used.Name("requests.storage", resource.BinarySI)
+		if err = r.syncResource(ctx, pod, "", "storage", storage, *podController); err != nil {
+			r.Logger.Error(err, "syncMeteringQuota failed")
+		}
+	}
+
+	if err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		if err := r.Get(ctx, types.NamespacedName{Namespace: podController.Namespace, Name: podController.Name}, podController); err != nil {
+			return client.IgnoreNotFound(err)
+		}
+		podController.Status.LatestUpdateTime = time.Now().Unix()
+		podController.Status.SeqID++
+		if err = r.Status().Update(ctx, podController); err != nil {
+			return err
+		}
+		return nil
+	}); err != nil {
+		r.Logger.Error(err, "UpdateResourceUsed failed")
+		return fmt.Errorf("pod controller update err:%v", err)
+	}
+	r.Logger.Info("pod controller calculate resource success")
+	return nil
+}
+
+func (r *PodResourceReconciler) checkPodStatus(pod v1.Pod) bool {
+	if pod.Status.Phase == v1.PodRunning {
+		return true
+	}
+	r.Logger.Info("pod status is  not ready", "pod name", pod.Name, "pod namespace", pod.Namespace, "pod status", pod.Status.Phase)
+	return false
+}
+
+func (r *PodResourceReconciler) checkPodNamespace(pod v1.Pod) bool {
+	ns := v1.Namespace{}
+	if err := r.Get(context.Background(), client.ObjectKey{Name: pod.Namespace}, &ns); err != nil {
+		r.Logger.Error(err, "get namespace failed", "pod NS", pod.Namespace)
+		return false
+	}
+	if _, ok := ns.Annotations[userv1.UserAnnotationOwnerKey]; !ok {
+		r.Logger.Info("pod namespace is not user namespace", "pod name", pod.Name, "pod namespace", pod.Namespace)
+		return false
+	}
+	return true
+}
+
+func (r *PodResourceReconciler) syncResource(ctx context.Context, pod v1.Pod, containerName string, resourceName v1.ResourceName, Used *resource.Quantity, podController meteringv1.PodResource) error {
+	podResource := meteringv1.Resource{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      GetResourceName(pod.Namespace, pod.Name, containerName, resourceName, podController.Status.SeqID),
+			Namespace: r.MeteringSystemNameSpace,
+		},
+	}
+
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &podResource, func() error {
+		if podResource.Spec.Resources == nil {
+			podResource.Spec.Resources = make(map[v1.ResourceName]meteringv1.ResourceInfo)
+		}
+		podResource.Spec.Resources[resourceName] = meteringv1.ResourceInfo{
+			Used:      Used,
+			TimeStamp: time.Now().Unix(),
+			NameSpace: pod.Namespace,
+		}
+		return nil
+	}); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (r *PodResourceReconciler) checkResourceExist(resourceName v1.ResourceName, container v1.Container) (resource.Quantity, bool) {
+	//r.Logger.V(1).Info("pod container", "resourceName", resourceName, "container", container)
+	if _, ok := container.Resources.Limits[resourceName]; ok {
+		return container.Resources.Limits[resourceName], ok
+		//r.Logger.Info("container.Resources.Limits resource not available", "resource name", resourceName)
+	} else if _, ok := container.Resources.Requests[resourceName]; ok {
+		return container.Resources.Requests[resourceName], ok
+	}
+
+	return resource.Quantity{}, false
+}
+
+func GetResourceName(namespaceName, podName string, containerName string, resourceName v1.ResourceName, seqID int64) string {
+	if containerName == "" {
+		return fmt.Sprintf("%s-%s-%s-%s-%v", namespaceName, meteringv1.PodResourcePricePrefix, podName, resourceName, seqID)
+	}
+
+	return fmt.Sprintf("%s-%s-%s-%s-%s-%v", namespaceName, meteringv1.PodResourcePricePrefix, podName, containerName, resourceName, seqID)
+}
+
+// SetupWithManager sets up the controller with the Manager.
+func (r *PodResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	const controllerName = "podResource-controller"
+	r.Logger = ctrl.Log.WithName(controllerName)
+	r.Logger.V(1).Info("reconcile podResource-controller")
+	r.MeteringSystemNameSpace = os.Getenv(meteringv1.METERINGNAMESPACEENV)
+	if os.Getenv(meteringv1.METERINGNAMESPACEENV) == "" {
+		r.MeteringSystemNameSpace = meteringv1.DEFAULTMETERINGNAMESPACE
+	}
+	return ctrl.NewControllerManagedBy(mgr).
+		For(&meteringv1.PodResource{}).
+		Complete(r)
+}
@@ -7,6 +7,81 @@ metadata:
 ---
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.8.0
+  creationTimestamp: null
+  name: extensionresourceprices.metering.sealos.io
+spec:
+  group: metering.sealos.io
+  names:
+    kind: ExtensionResourcePrice
+    listKind: ExtensionResourcePriceList
+    plural: extensionresourceprices
+    singular: extensionresourceprice
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: ExtensionResourcePrice is the Schema for the extensionresourceprices
+          API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ExtensionResourcePriceSpec defines the desired state of ExtensionResourcePrice
+            properties:
+              resourceName:
+                type: string
+              resources:
+                additionalProperties:
+                  properties:
+                    describe:
+                      type: string
+                    price:
+                      format: int64
+                      type: integer
+                    unit:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  required:
+                  - price
+                  - unit
+                  type: object
+                type: object
+            type: object
+          status:
+            description: ExtensionResourcePriceStatus defines the observed state of
+              ExtensionResourcePrice
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
 metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.8.0
@@ -50,6 +125,7 @@ spec:
          metadata:
            type: object
          spec:
+            description: MeteringSpec defines the desired state of Metering
            properties:
              namespace:
                type: string
@@ -57,18 +133,38 @@ spec:
                type: string
              resources:
                additionalProperties:
-                  anyOf:
-                  - type: integer
-                  - type: string
-                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
-                  x-kubernetes-int-or-string: true
-                description: ResourceList is a set of (resource name, quantity) pairs.
+                  properties:
+                    describe:
+                      type: string
+                    price:
+                      format: int64
+                      type: integer
+                    unit:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                    used:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  required:
+                  - price
+                  - unit
+                  type: object
                type: object
+              timeInterval:
+                description: timeInterval unit is minutes
+                type: integer
            required:
            - namespace
            - owner
            type: object
          status:
+            description: MeteringStatus defines the observed state of Metering
            properties:
              billingListD:
                items:
@@ -100,27 +196,12 @@ spec:
                      type: integer
                  type: object
                type: array
-              billingListM:
-                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
-                  of cluster Important: Run "make" to regenerate code after modifying
-                  this file'
-                items:
-                  properties:
-                    amount:
-                      format: int64
-                      type: integer
-                    settled:
-                      type: boolean
-                    timeInterval:
-                      type: string
-                    timeStamp:
-                      format: int64
-                      type: integer
-                  type: object
-                type: array
              latestUpdateTime:
                format: int64
                type: integer
+              seqID:
+                format: int64
+                type: integer
              totalAmount:
                format: int64
                type: integer
@@ -137,6 +218,162 @@ status:
  conditions: []
  storedVersions: []
 ---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.8.0
+  creationTimestamp: null
+  name: podresources.metering.sealos.io
+spec:
+  group: metering.sealos.io
+  names:
+    kind: PodResource
+    listKind: PodResourceList
+    plural: podresources
+    singular: podresource
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: PodResource is the Schema for the podresources API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: PodResourceSpec defines the desired state of PodResource
+            properties:
+              interval:
+                default: 50
+                description: update used resources every Interval minutes
+                minimum: 1
+                type: integer
+              resourceName:
+                type: string
+              resources:
+                additionalProperties:
+                  properties:
+                    describe:
+                      type: string
+                    price:
+                      format: int64
+                      type: integer
+                    unit:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  required:
+                  - price
+                  - unit
+                  type: object
+                type: object
+            type: object
+          status:
+            description: PodResourceStatus defines the observed state of PodResource
+            properties:
+              latestUpdateTime:
+                format: int64
+                type: integer
+              seqID:
+                format: int64
+                type: integer
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.8.0
+  creationTimestamp: null
+  name: resources.metering.sealos.io
+spec:
+  group: metering.sealos.io
+  names:
+    kind: Resource
+    listKind: ResourceList
+    plural: resources
+    singular: resource
+  scope: Namespaced
+  versions:
+  - name: v1
+    schema:
+      openAPIV3Schema:
+        description: Resource is the Schema for the resources API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ResourceSpec defines the desired state of Resource
+            properties:
+              resources:
+                additionalProperties:
+                  properties:
+                    namespace:
+                      type: string
+                    time:
+                      format: int64
+                      type: integer
+                    used:
+                      anyOf:
+                      - type: integer
+                      - type: string
+                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
+                      x-kubernetes-int-or-string: true
+                  type: object
+                type: object
+            type: object
+          status:
+            description: ResourceStatus defines the observed state of Resource
+            properties:
+              status:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
+status:
+  acceptedNames:
+    kind: ""
+    plural: ""
+  conditions: []
+  storedVersions: []
+---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
@@ -187,6 +424,66 @@ metadata:
  creationTimestamp: null
  name: metering-manager-role
 rules:
+- apiGroups:
+  - ""
+  resources:
+  - namespaces
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - pods
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - resourcequotas
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - infra.sealos.io
+  resources:
+  - infras
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - extensionresourceprices
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
 - apiGroups:
  - metering.sealos.io
  resources:
@@ -213,6 +510,56 @@ rules:
  - get
  - patch
  - update
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources/finalizers
+  verbs:
+  - update
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - podresources/status
+  verbs:
+  - get
+  - patch
+  - update
+- apiGroups:
+  - metering.sealos.io
+  resources:
+  - resources
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
@@ -368,12 +715,13 @@ spec:
        command:
        - /manager
        env:
-        - name: SEALOS_SYSTEM_NAMESPACE
+        - name: METERING_SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: ghcr.io/labring/sealos-metering-controller:dev
+        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
@@ -21,9 +21,10 @@ import (

 	infrav1 "github.com/labring/sealos/controllers/infra/api/v1"

-	"github.com/labring/sealos/controllers/metering/controllers"
 	userv1 "github.com/labring/sealos/controllers/user/api/v1"

+	"github.com/labring/sealos/controllers/metering/controllers"
+
 	"os"

 	// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
@@ -90,6 +91,14 @@ func main() {
 		setupLog.Error(err, "unable to create controller", "controller", "Metering")
 		os.Exit(1)
 	}
+	if err = (&controllers.PodResourceReconciler{
+		Client: mgr.GetClient(),
+		Scheme: mgr.GetScheme(),
+	}).SetupWithManager(mgr); err != nil {
+		setupLog.Error(err, "unable to create controller", "controller", "PodResource")
+		os.Exit(1)
+	}
+
 	//+kubebuilder:scaffold:builder

 	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
@@ -0,0 +1,41 @@
+package api
+
+import (
+	"fmt"
+	userv1 "github.com/labring/sealos/controllers/user/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+	"time"
+)
+
+const AccountBalanceYaml = `
+apiVersion: user.sealos.io/v1
+kind: AccountBalance
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+`
+
+func GetAccountBalance(namespace string, name string) (*userv1.AccountBalance, error) {
+	gvr := userv1.GroupVersion.WithResource("accountbalances")
+	var accountbalance userv1.AccountBalance
+	if err := baseapi.GetObject(namespace, name, gvr, &accountbalance); err != nil {
+		return nil, err
+	}
+	return &accountbalance, nil
+}
+
+func EnsureAccountBalanceCreate(namespace string, name string, times int) (*userv1.AccountBalance, error) {
+	time.Sleep(time.Second)
+	for i := 1; i <= times; i++ {
+		accountBalance, err := GetAccountBalance(namespace, name)
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+		if accountBalance.Spec.Amount > 0 {
+			return accountBalance, nil
+		}
+		time.Sleep(time.Second)
+	}
+	return nil, fmt.Errorf("accountbalance get faile")
+}
@@ -0,0 +1,25 @@
+package api
+
+import (
+	userv1 "github.com/labring/sealos/controllers/user/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+)
+
+const AccountYaml = `
+apiVersion: user.sealos.io/v1
+kind: Account
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+
+
+`
+
+func GetAccount(namespace string, name string) (*userv1.Account, error) {
+	gvr := userv1.GroupVersion.WithResource("accounts")
+	var account userv1.Account
+	if err := baseapi.GetObject(namespace, name, gvr, &account); err != nil {
+		return nil, err
+	}
+	return &account, nil
+}
@@ -0,0 +1,42 @@
+package api
+
+import (
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+)
+
+const ExtensionResourcePriceYaml = `
+apiVersion: metering.sealos.io/v1
+kind: ExtensionResourcePrice
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+
+spec:
+  resourceName: pod
+  resources:
+    cpu:
+      unit: "1"
+      price: 1
+      describe: "cost per cpu per hour（price:100 = 1¥）"
+`
+
+func GetExtensionResourcePrice(namespace string, name string) (*meteringv1.ExtensionResourcePrice, error) {
+	gvr := meteringv1.GroupVersion.WithResource("extensionresourceprices")
+	var extensionResourcesPrice meteringv1.ExtensionResourcePrice
+	if err := baseapi.GetObject(namespace, name, gvr, &extensionResourcesPrice); err != nil {
+		return nil, err
+	}
+	return &extensionResourcesPrice, nil
+}
+
+func DeleteExtensionResourcePrice(namespace string, name string) error {
+	_, err := baseapi.KubeDeleteFromTemplate(ExtensionResourcePriceYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+	if err != nil {
+		return err
+	}
+	return nil
+}
@@ -0,0 +1,65 @@
+package api
+
+import (
+	"fmt"
+	infrav1 "github.com/labring/sealos/controllers/infra/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+	"time"
+)
+
+const InfraYaml = `
+apiVersion: infra.sealos.io/v1
+kind: Infra
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+spec:
+  hosts:
+  - roles: [master] 
+    count: 1
+    flavor: t2.medium
+    image: "ami-0d66b970b9f16f1f5"
+    disks:
+    - capacity: 16
+      volumeType: gp3
+      # allowed value is root|data
+      type: "root"
+
+  - roles: [ node ] 
+    count: 1 
+    flavor: t2.medium
+    image: "ami-0d66b970b9f16f1f5"
+    disks:
+    - capacity: 16
+      volumeType: gp3
+      # allowed value is root|data
+      type: "root"
+`
+
+func GetInfra(namespace string, name string) (*infrav1.Infra, error) {
+	gvr := infrav1.GroupVersion.WithResource("infras")
+	var infra infrav1.Infra
+	if err := baseapi.GetObject(namespace, name, gvr, &infra); err != nil {
+		return nil, err
+	}
+	return &infra, nil
+}
+
+func EnsureInfra(namespace string, name string, times int) error {
+	_, err := GetInfra(namespace, name)
+	if err != nil {
+		baseapi.CreateCRD(namespace, name, InfraYaml)
+	}
+
+	for i := 0; i < times; i++ {
+		infra, err := GetInfra(namespace, name)
+		if err != nil {
+			continue
+		}
+		if infra.Status.Status == infrav1.Running.String() {
+			return nil
+		}
+		time.Sleep(time.Second)
+	}
+	return fmt.Errorf("infra time out")
+}
@@ -0,0 +1,82 @@
+package api
+
+import (
+	"fmt"
+	"time"
+
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+)
+
+const MeteringYaml = `
+apiVersion: metering.sealos.io/v1
+kind: Metering
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+
+
+`
+
+func GetMetering(namespace string, name string) (*meteringv1.Metering, error) {
+	gvr := meteringv1.GroupVersion.WithResource("meterings")
+
+	var metering meteringv1.Metering
+	if err := baseapi.GetObject(namespace, name, gvr, &metering); err != nil {
+		return nil, err
+	}
+	return &metering, nil
+}
+
+func EnsureMetering(namespace string, name string) {
+	if _, err := GetMetering(namespace, name); err != nil {
+		baseapi.CreateCRD(namespace, name, MeteringYaml)
+	}
+}
+
+func EnsureMeteringCalculate(namespace string, name string, times int) (*meteringv1.Metering, error) {
+	EnsureMetering(namespace, name)
+	time.Sleep(time.Second)
+	for i := 1; i <= times; i++ {
+		metering, err := GetMetering(namespace, name)
+		if err != nil {
+			return nil, err
+		}
+		if metering.Status.TotalAmount > 0 {
+			return metering, nil
+		}
+		time.Sleep(time.Second)
+	}
+	return nil, fmt.Errorf("metering calculate failed")
+}
+
+func DeleteMetering(namespace string, name string) error {
+	_, err := baseapi.KubeDeleteFromTemplate(MeteringYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func EnsureMeteringUsed(namespace string, name string, times int) (*meteringv1.Metering, error) {
+	EnsureMetering(namespace, name)
+	time.Sleep(time.Second)
+	for i := 1; i <= times; i++ {
+		metering, err := GetMetering(namespace, name)
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+		if _, ok := metering.Spec.Resources["cpu"]; !ok {
+			return nil, fmt.Errorf("metering resource cpu is not found")
+		}
+		if metering.Spec.Resources["cpu"].Used.Value() > 0 {
+			return metering, nil
+		}
+		time.Sleep(time.Second)
+	}
+	return nil, fmt.Errorf("metering calculate failed")
+}
@@ -0,0 +1,156 @@
+package api
+
+import (
+	"context"
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const PodControllerYaml = `
+---
+apiVersion: metering.sealos.io/v1
+kind: PodResource
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+spec:
+  interval: 1
+  resources:
+    cpu:
+      unit: "1"
+      price: 1
+      describe: "cost per cpu per hour（price:100 = 1¥）"
+
+    storage:
+      unit: "1G"
+      price:  1
+      describe: "cost per gigabyte of storage per hour（price:100 = 1¥）"
+
+`
+
+func CreatePodController(namespace string, name string) {
+	baseapi.MustKubeApplyFromTemplate(PodControllerYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+}
+
+func DeletePodController(namespace string, name string) error {
+	_, err := baseapi.KubeDeleteFromTemplate(PodControllerYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func GetPodController(namespace string, name string) (*meteringv1.PodResource, error) {
+	gvr := meteringv1.GroupVersion.WithResource("podresourceprices")
+	var podController meteringv1.PodResource
+	if err := baseapi.GetObject(namespace, name, gvr, &podController); err != nil {
+		return nil, err
+	}
+	return &podController, nil
+}
+
+func EnsurePodController(namespace, name string) {
+	if _, err := GetPodController(namespace, name); err != nil {
+		CreatePodController(namespace, name)
+		return
+	}
+}
+
+const PodYaml = `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+spec:
+  containers:
+    - name: nginx
+      image: nginx:1.14.2
+      resources:
+        requests:
+          cpu: 1000m
+          memory: 1Gi
+          ephemeral-storage: 1Gi
+
+        limits:
+          cpu: 1000m
+          memory: 1Gi
+          ephemeral-storage: 1Gi
+      ports:
+        - containerPort: 80
+  volumes:
+  - name: task-pv-storage
+    persistentVolumeClaim:
+      claimName: task-pv-claim
+
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: task-pv-volume
+  namespace: ${namespace}
+  labels:
+    type: local
+spec:
+  storageClassName: manual
+  capacity:
+    storage: 1Gi
+  accessModes:
+    - ReadWriteOnce
+  hostPath:
+    path: "/"
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: task-pv-claim
+  namespace: ${namespace}
+spec:
+  storageClassName: manual
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+`
+
+func CreatPod(namespace string, name string) {
+	baseapi.MustKubeApplyFromTemplate(PodYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+}
+
+func DeletePod(namespace string, name string) error {
+	_, err := baseapi.KubeDeleteFromTemplate(PodYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+	if err != nil {
+		return err
+	}
+	return nil
+}
+
+func EnsurePod(namespace string, name string) {
+	client := baseapi.GetDefaultKubernetesClient()
+	_, err := client.CoreV1().Pods(namespace).Get(context.TODO(), name, metav1.GetOptions{})
+	if err != nil {
+		CreatPod(namespace, name)
+		return
+	}
+}
+
+func GetPod(namespace string, name string) (*v1.Pod, error) {
+	client := baseapi.GetDefaultKubernetesClient()
+	pod, err := client.CoreV1().Pods(namespace).Get(context.TODO(), name, metav1.GetOptions{})
+	return pod, err
+}
@@ -0,0 +1,59 @@
+package api
+
+import (
+	"fmt"
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+	"time"
+)
+
+const ResourceYaml = `
+apiVersion: metering.sealos.io/v1
+kind: Resource
+metadata:
+  name: ${name}
+  namespace: ${namespace}
+spec:
+  resources:
+    resourceName: "cpu"
+    Used: 1
+`
+
+func GetResource(namespace string, name string) (*meteringv1.Resource, error) {
+	gvr := meteringv1.GroupVersion.WithResource("resources")
+	var resource meteringv1.Resource
+	if err := baseapi.GetObject(namespace, name, gvr, &resource); err != nil {
+		return nil, err
+	}
+	return &resource, nil
+}
+
+func EnsureResourceCreate(namespace string, name string, times int) (*meteringv1.Resource, error) {
+	time.Sleep(time.Second)
+	for i := 1; i <= times; i++ {
+		resource, err := GetResource(namespace, name)
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+		if _, ok := resource.Spec.Resources["cpu"]; !ok {
+			return nil, fmt.Errorf("not fount cpu resource used ")
+		}
+		if resource.Spec.Resources["cpu"].Used.Value() > 0 {
+			return resource, nil
+		}
+		time.Sleep(time.Second)
+	}
+	return nil, fmt.Errorf("resource create failed")
+}
+
+func DeleteResource(namespace string, name string) error {
+	_, err := baseapi.KubeDeleteFromTemplate(ResourceYaml, map[string]string{
+		"namespace": namespace,
+		"name":      name,
+	})
+	if err != nil {
+		return err
+	}
+	return nil
+}
@@ -0,0 +1,267 @@
+package e2e
+
+import (
+	"fmt"
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	"github.com/labring/sealos/controllers/metering/controllers"
+	"github.com/labring/sealos/controllers/metering/testdata/api"
+	userv1 "github.com/labring/sealos/controllers/user/api/v1"
+	baseapi "github.com/labring/sealos/test/testdata/api"
+	"log"
+	"os"
+	"testing"
+	"time"
+)
+
+const (
+	TestNamespace    = "ns-metering-test"
+	PodName          = "nginx-test"
+	DefaultOwner     = "metering-test"
+	AccountNamespace = "sealos-system"
+	InfraName        = "yyj-test-1"
+	containerName    = "nginx"
+)
+
+var MeteringSystemNamespace string
+
+func init() {
+	MeteringSystemNamespace = os.Getenv(meteringv1.METERINGNAMESPACEENV)
+	if MeteringSystemNamespace == "" {
+		MeteringSystemNamespace = "metering-system"
+	}
+	baseapi.EnsureNamespace(MeteringSystemNamespace)
+	// first export METERING_INTERVAL=1，then run test
+	fmt.Println("METERING_INTERVAL:", os.Getenv("METERING_INTERVAL"))
+}
+
+func TestMetering(t *testing.T) {
+	t.Run("metering should be ok", func(t *testing.T) {
+		t.Run("metering should be created when create a user ns", func(t *testing.T) {
+			t.Log("create metering  ")
+			baseapi.EnsureNamespace(TestNamespace)
+			time.Sleep(time.Second * 3)
+
+			t.Log("ensure metering is created")
+			_, err := api.GetMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+			if err != nil {
+				t.Fatalf("failed to get metering: %v", err)
+			}
+
+			t.Log("ensure metering is delete after delete namespace")
+			err = baseapi.DeleteNamespace(TestNamespace)
+			if err != nil {
+				t.Fatalf("failed to delete namespace: %v", err)
+			}
+			time.Sleep(time.Second)
+			_, err = api.GetMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+			if err == nil {
+				t.Fatalf("success get metering: %v", err)
+			}
+		})
+
+		t.Run("extension should be ok", func(t *testing.T) {
+			// test extension will register cpu price to metering
+			t.Log("create metering  ")
+			baseapi.EnsureNamespace(TestNamespace)
+			time.Sleep(time.Second * 3)
+			metering, err := api.GetMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+			if err != nil {
+				t.Fatalf("success get metering: %v", err)
+			}
+
+			if _, ok := metering.Spec.Resources["cpu"]; ok {
+				t.Fatalf("metering spec.Resources should not have cpu")
+			}
+			t.Log("create extensionResourcePrice  ")
+			baseapi.CreateCRD(MeteringSystemNamespace, meteringv1.GetExtensionResourcePriceName(meteringv1.PodResourcePricePrefix), api.ExtensionResourcePriceYaml)
+			time.Sleep(time.Second)
+			metering, err = api.GetMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+			if err != nil {
+				t.Fatalf("success get metering: %v", err)
+			}
+
+			if _, ok := metering.Spec.Resources["cpu"]; !ok {
+				t.Fatalf("metering spec.Resources should  have cpu")
+			}
+		})
+
+		t.Run("pod controller should be ok", func(t *testing.T) {
+			baseapi.EnsureNamespace(TestNamespace)
+			time.Sleep(5 * time.Second)
+			t.Log("creat pod controller")
+			api.CreatePodController(MeteringSystemNamespace, meteringv1.PodResourcePricePrefix)
+
+			time.Sleep(5 * time.Second)
+			t.Log("ensure extension resource is created")
+			podExtensionResourcePrice, err := api.GetExtensionResourcePrice(MeteringSystemNamespace, meteringv1.GetExtensionResourcePriceName(meteringv1.PodResourcePricePrefix))
+			if err != nil {
+				t.Fatalf("failed to get extension resource: %v", err)
+			}
+
+			if _, ok := podExtensionResourcePrice.Spec.Resources["cpu"]; !ok {
+				t.Fatalf("failed to get cpu price,resource info:%+v", podExtensionResourcePrice.Spec.Resources)
+			}
+			if _, ok := podExtensionResourcePrice.Spec.Resources["storage"]; !ok {
+				t.Fatalf("failed to get storage price,resource info:%+v", podExtensionResourcePrice.Spec.Resources)
+			}
+
+			t.Log("create a nginx pod")
+			api.CreatPod(TestNamespace, PodName)
+
+			t.Log("ensure resource CR is created")
+			resource, err := api.EnsureResourceCreate(MeteringSystemNamespace, controllers.GetResourceName(TestNamespace, PodName, containerName, "cpu", 1), 90)
+			if err != nil {
+				resource, err = api.EnsureResourceCreate(MeteringSystemNamespace, controllers.GetResourceName(TestNamespace, PodName, containerName, "cpu", 2), 90)
+				if err != nil {
+					t.Fatalf(err.Error())
+				}
+			}
+
+			if _, ok := resource.Spec.Resources["cpu"]; !ok {
+				t.Fatalf("not fount cpu resource used ")
+			}
+			if resource.Spec.Resources["cpu"].Used.Value() != 1 {
+				t.Fatalf("not fount cpu resource used %v", resource.Spec.Resources["cpu"].Used.Value())
+			}
+			t.Log(resource)
+
+		})
+
+		t.Run("metering used update and calculate should be ok", func(t *testing.T) {
+			api.EnsurePodController(MeteringSystemNamespace, meteringv1.PodResourcePricePrefix)
+			time.Sleep(time.Second * 5)
+			baseapi.EnsureNamespace(TestNamespace)
+			api.EnsurePod(TestNamespace, PodName)
+			time.Sleep(time.Second * 5)
+
+			metering, err := api.GetMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+			if err != nil {
+				t.Fatalf("fail get metering: %v", err)
+			}
+
+			if _, ok := metering.Spec.Resources["cpu"]; !ok {
+				t.Fatalf("not fount cpu resource info in metering ")
+			}
+
+			t.Log("metering calculate should be ok")
+			metering, err = api.EnsureMeteringCalculate(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace, 90)
+			if err != nil {
+				t.Fatalf("metering calculate failer: %v,calculate: %v", err, metering)
+			}
+		})
+
+		t.Run("metering create accountBalance should be ok", func(t *testing.T) {
+			api.EnsurePodController(MeteringSystemNamespace, meteringv1.PodResourcePricePrefix)
+			time.Sleep(time.Second * 2)
+			baseapi.EnsureNamespace(TestNamespace)
+			api.EnsurePod(TestNamespace, PodName)
+			time.Sleep(time.Second * 2)
+
+			t.Log("ensure accountBalance is created")
+			metering, err := api.GetMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+			if err != nil {
+				t.Fatalf("fail get metering: %v", err)
+			}
+			accountBalance, err := api.EnsureAccountBalanceCreate(MeteringSystemNamespace, fmt.Sprintf("%s-%s-%v", userv1.AccountBalancePrefix, metering.Spec.Owner, 1), 90)
+			if err != nil {
+				t.Log("ensure accountBalance is created again")
+				accountBalance, err = api.EnsureAccountBalanceCreate(MeteringSystemNamespace, fmt.Sprintf("%s-%s-%v", userv1.AccountBalancePrefix, metering.Spec.Owner, 2), 90)
+				if err != nil {
+					t.Fatalf("failed to create accountBalance: %v", err)
+				}
+			}
+
+			if accountBalance.Spec.Amount == 0 || accountBalance.Spec.Owner == "" {
+				t.Fatalf("failed to create accountBalance: %v", accountBalance)
+			}
+
+			time.Sleep(2 * time.Second)
+			account, err := api.GetAccount(AccountNamespace, metering.Spec.Owner)
+			if err != nil {
+				t.Fatalf("fail get account: %v", err)
+			}
+
+			if account.Status.DeductionBalance == 0 {
+				t.Fatalf("account fail deduction: %v", err)
+			}
+
+			defer t.Log(fmt.Sprintf("account:%v", account))
+		})
+
+		t.Run("metering calculate infra should be ok", func(t *testing.T) {
+			t.Log("ensure infra is created")
+			baseapi.EnsureNamespace(TestNamespace)
+			baseapi.CreateCRD(TestNamespace, InfraName, api.InfraYaml)
+			err := api.EnsureInfra(TestNamespace, InfraName, 12)
+			//if err != nil {
+			//	t.Fatalf(err.Error())
+			//}
+			infra, err := api.GetInfra(TestNamespace, InfraName)
+			if err != nil {
+				t.Fatalf("failed to get infra: %v", err)
+			}
+
+			t.Log("check infra QueryPrice")
+			price, err := infra.QueryPrice()
+			t.Log("infra price is ", price)
+			if err != nil {
+				t.Fatalf("failed to query infra price: %v", err)
+			}
+		})
+	})
+	t.Cleanup(clear)
+}
+
+func clear() {
+	//time.Sleep(200 * time.Second)
+	err := baseapi.DeleteNamespace(TestNamespace)
+	if err != nil {
+		log.Println(err)
+	}
+
+	err = api.DeleteMetering(MeteringSystemNamespace, meteringv1.MeteringPrefix+TestNamespace)
+	if err != nil {
+		log.Println(err)
+	}
+
+	err = baseapi.DeleteCRD(MeteringSystemNamespace, meteringv1.GetExtensionResourcePriceName(meteringv1.PodResourcePricePrefix), api.ExtensionResourcePriceYaml)
+	if err != nil {
+		log.Println(err)
+	}
+
+	err = api.DeletePodController(MeteringSystemNamespace, meteringv1.PodResourcePricePrefix)
+	if err != nil {
+		log.Println(err)
+	}
+
+	err = api.DeletePod(TestNamespace, PodName)
+	if err != nil {
+		log.Println(err)
+	}
+
+	if err = api.DeleteExtensionResourcePrice(MeteringSystemNamespace, meteringv1.PodResourcePricePrefix); err != nil {
+		log.Println(err)
+	}
+
+	if err = baseapi.DeleteCRD(AccountNamespace, DefaultOwner, api.AccountYaml); err != nil {
+		log.Println(err)
+	}
+
+	if err = baseapi.DeleteCRD(TestNamespace, InfraName, api.InfraYaml); err != nil {
+		log.Println(err)
+	}
+
+	for i := 0; i <= 5; i++ {
+		if err = baseapi.DeleteCRD(MeteringSystemNamespace, controllers.GetResourceName(TestNamespace, PodName, containerName, "cpu", int64(i)), api.ResourceYaml); err != nil {
+			log.Println(err)
+		}
+
+		if err = baseapi.DeleteCRD(MeteringSystemNamespace, controllers.GetResourceName(TestNamespace, PodName, "", "storage", int64(i)), api.ResourceYaml); err != nil {
+			log.Println(err)
+		}
+
+		if err = baseapi.DeleteCRD(MeteringSystemNamespace, fmt.Sprintf("%s-%s-%v", userv1.AccountBalancePrefix, DefaultOwner, i), api.AccountBalanceYaml); err != nil {
+			log.Println(err)
+		}
+	}
+}
@@ -58,4 +58,12 @@ resources:
  kind: Payment
  path: github.com/labring/sealos/controllers/user/api/v1
  version: v1
+- api:
+    crdVersion: v1
+    namespaced: true
+  domain: sealos.io
+  group: user
+  kind: AccountBalance
+  path: github.com/labring/sealos/controllers/user/api/v1
+  version: v1
 version: "3"
@@ -44,7 +44,12 @@ Query the Charging status Order, and get it status, if success, add the Amount t
 // namespace create by the user.

 type Charge struct {
-	Amount  int64       `json:"balance,omitempty"`
+	Amount int64 `json:"balance,omitempty"`
+
+	// deduction info will Record in the Charge
+	DeductionAmount    int64  `json:"deductionAmount,omitempty"`
+	AccountBalanceName string `json:"accountBalanceName"`
+
 	Time    metav1.Time `json:"time,omitempty"`
 	Status  string      `json:"status,omitempty"`
 	TradeNO string      `json:"tradeNO,omitempty"`
@@ -55,10 +60,12 @@ type AccountSpec struct{}

 // AccountStatus defines the observed state of Account
 type AccountStatus struct {
+	// Recharge amount
 	Balance int64 `json:"balance,omitempty"`
-	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
-	// Important: Run "make" to regenerate code after modifying this file
-	ChargeList []Charge `json:"chargeList,omitempty"`
+
+	//Deduction amount
+	DeductionBalance int64    `json:"deductionBalance,omitempty"`
+	ChargeList       []Charge `json:"chargeList,omitempty"`
 }

 //+kubebuilder:object:root=true
@@ -0,0 +1,62 @@
+/*
+Copyright 2022 labring.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1
+
+import (
+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const AccountBalancePrefix = "accountbalance"
+
+// AccountBalanceSpec defines the desired state of AccountBalance
+type AccountBalanceSpec struct {
+	Owner     string `json:"owner"`
+	TimeStamp int64  `json:"timeStamp,omitempty"`
+	Amount    int64  `json:"amount,omitempty"`
+}
+
+// AccountBalanceStatus defines the observed state of AccountBalance
+type AccountBalanceStatus struct {
+	Status meteringv1.Status `json:"status,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+//+kubebuilder:subresource:status
+//+kubebuilder:printcolumn:name="Amount",type=integer,JSONPath=".spec.amount"
+
+// AccountBalance is the Schema for the accountbalances API
+type AccountBalance struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec   AccountBalanceSpec   `json:"spec,omitempty"`
+	Status AccountBalanceStatus `json:"status,omitempty"`
+}
+
+//+kubebuilder:object:root=true
+
+// AccountBalanceList contains a list of AccountBalance
+type AccountBalanceList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []AccountBalance `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&AccountBalance{}, &AccountBalanceList{})
+}
@@ -52,6 +52,95 @@ func (in *Account) DeepCopyObject() runtime.Object {
 	return nil
 }

+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *AccountBalance) DeepCopyInto(out *AccountBalance) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	out.Spec = in.Spec
+	out.Status = in.Status
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AccountBalance.
+func (in *AccountBalance) DeepCopy() *AccountBalance {
+	if in == nil {
+		return nil
+	}
+	out := new(AccountBalance)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *AccountBalance) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *AccountBalanceList) DeepCopyInto(out *AccountBalanceList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]AccountBalance, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AccountBalanceList.
+func (in *AccountBalanceList) DeepCopy() *AccountBalanceList {
+	if in == nil {
+		return nil
+	}
+	out := new(AccountBalanceList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *AccountBalanceList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *AccountBalanceSpec) DeepCopyInto(out *AccountBalanceSpec) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AccountBalanceSpec.
+func (in *AccountBalanceSpec) DeepCopy() *AccountBalanceSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(AccountBalanceSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *AccountBalanceStatus) DeepCopyInto(out *AccountBalanceStatus) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AccountBalanceStatus.
+func (in *AccountBalanceStatus) DeepCopy() *AccountBalanceStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(AccountBalanceStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *AccountList) DeepCopyInto(out *AccountList) {
 	*out = *in
@@ -0,0 +1,63 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.10.0
+  creationTimestamp: null
+  name: accountbalances.user.sealos.io
+spec:
+  group: user.sealos.io
+  names:
+    kind: AccountBalance
+    listKind: AccountBalanceList
+    plural: accountbalances
+    singular: accountbalance
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.amount
+      name: Amount
+      type: integer
+    name: v1
+    schema:
+      openAPIV3Schema:
+        description: AccountBalance is the Schema for the accountbalances API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AccountBalanceSpec defines the desired state of AccountBalance
+            properties:
+              amount:
+                format: int64
+                type: integer
+              owner:
+                type: string
+              timeStamp:
+                format: int64
+                type: integer
+            required:
+            - owner
+            type: object
+          status:
+            description: AccountBalanceStatus defines the observed state of AccountBalance
+            properties:
+              status:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
@@ -39,12 +39,10 @@ spec:
            description: AccountStatus defines the observed state of Account
            properties:
              balance:
+                description: Recharge amount
                format: int64
                type: integer
              chargeList:
-                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
-                  of cluster Important: Run "make" to regenerate code after modifying
-                  this file'
                items:
                  properties:
                    balance:
@@ -59,6 +57,10 @@ spec:
                      type: string
                  type: object
                type: array
+              deductionBalance:
+                description: Deduction amount
+                format: int64
+                type: integer
            type: object
        type: object
    served: true
@@ -7,6 +7,7 @@ resources:
 - bases/user.sealos.io_usergroupbindings.yaml
 - bases/user.sealos.io_accounts.yaml
 - bases/user.sealos.io_payments.yaml
+- bases/user.sealos.io_accountbalances.yaml
 #+kubebuilder:scaffold:crdkustomizeresource

 patchesStrategicMerge:
@@ -20,6 +21,7 @@ patchesStrategicMerge:
 #- patches/webhook_in_usergroupbindings.yaml
 #- patches/webhook_in_accounts.yaml
 #- patches/webhook_in_payments.yaml
+#- patches/webhook_in_accountbalances.yaml
 #+kubebuilder:scaffold:crdkustomizewebhookpatch

 # [CERTMANAGER] To enable cert-manager, uncomment all the sections with [CERTMANAGER] prefix.
@@ -32,6 +34,7 @@ patchesStrategicMerge:
 #- patches/cainjection_in_usergroupbindings.yaml
 #- patches/cainjection_in_accounts.yaml
 #- patches/cainjection_in_payments.yaml
+#- patches/cainjection_in_accountbalances.yaml
 #+kubebuilder:scaffold:crdkustomizecainjectionpatch

 # the following config is for teaching kustomize how to do kustomization for CRDs.
@@ -0,0 +1,7 @@
+# The following patch adds a directive for certmanager to inject CA into the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: accountbalances.user.sealos.io
@@ -0,0 +1,16 @@
+# The following patch enables a conversion webhook for the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: accountbalances.user.sealos.io
+spec:
+  conversion:
+    strategy: Webhook
+    webhook:
+      clientConfig:
+        service:
+          namespace: system
+          name: webhook-service
+          path: /convert
+      conversionReviewVersions:
+      - v1
@@ -48,3 +48,5 @@ spec:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
+          - name: ACCOUNT_NAMESPACE
+            value: "sealos-system"
@@ -0,0 +1,24 @@
+# permissions for end users to edit accountbalances.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: accountbalance-editor-role
+rules:
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances/status
+  verbs:
+  - get
@@ -0,0 +1,20 @@
+# permissions for end users to view accountbalances.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: accountbalance-viewer-role
+rules:
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances/status
+  verbs:
+  - get
@@ -127,6 +127,18 @@ rules:
  - patch
  - update
  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
 - apiGroups:
  - user.sealos.io
  resources:
@@ -0,0 +1,6 @@
+apiVersion: user.sealos.io/v1
+kind: AccountBalance
+metadata:
+  name: accountbalance-sample
+spec:
+  # TODO(user): Add fields here
@@ -19,8 +19,11 @@ package controllers
 import (
 	"context"
 	"fmt"
+	"os"
 	"time"

+	meteringv1 "github.com/labring/sealos/controllers/metering/api/v1"
+
 	"github.com/labring/sealos/controllers/user/controllers/helper"

 	"github.com/go-logr/logr"
@@ -31,11 +34,8 @@ import (

 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

-	"k8s.io/apimachinery/pkg/api/errors"
-
 	"github.com/labring/sealos/pkg/pay"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
-	"sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/controller-runtime/pkg/source"

 	userv1 "github.com/labring/sealos/controllers/user/api/v1"
@@ -44,41 +44,42 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )

+const ACCOUNTNAMESPACEENV = "ACCOUNT_NAMESPACE"
+const DEFAULTACCOUNTNAMESPACE = "sealos-system"
+
 // AccountReconciler reconciles a Account object
 type AccountReconciler struct {
 	client.Client
-	Scheme *runtime.Scheme
-	Logger logr.Logger
+	Scheme                 *runtime.Scheme
+	Logger                 logr.Logger
+	AccountSystemNameSpace string
 }

 //+kubebuilder:rbac:groups=user.sealos.io,resources=accounts,verbs=get;list;watch;create;update;patch;delete
 //+kubebuilder:rbac:groups=user.sealos.io,resources=accounts/status,verbs=get;update;patch
 //+kubebuilder:rbac:groups=user.sealos.io,resources=accounts/finalizers,verbs=update
+//+kubebuilder:rbac:groups=user.sealos.io,resources=accountbalances,verbs=get;list;watch;create;update;patch;delete

-// Reconcile is part of the main kubernetes reconciliation loop which aims to
-// move the current state of the cluster closer to the desired state.
-// TODO(user): Modify the Reconcile function to compare the state specified by
-// the Account object against the actual cluster state, and then
-// perform operations to make the cluster state reflect the state specified by
-// the user.
-//
-// For more details, check Reconcile and its Result here:
-// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.2/pkg/reconcile
 func (r *AccountReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
-	logger := log.FromContext(ctx)
 	//It should not stop the normal process for the failure to delete the payment
 	// delete payments that exist for more than 5 minutes
 	if err := r.DeletePayment(ctx); err != nil {
 		r.Logger.Error(err, "delete payment failed")
 	}

-	payment := &userv1.Payment{}
-	err := r.Get(ctx, client.ObjectKey{Namespace: req.Namespace, Name: req.Name}, payment)
-	if errors.IsNotFound(err) {
-		return ctrl.Result{}, nil
+	accountBalance := userv1.AccountBalance{}
+	if err := r.Get(ctx, req.NamespacedName, &accountBalance); err == nil {
+		if err := r.updateDeductionBalance(ctx, &accountBalance); err != nil {
+			r.Logger.Error(err, err.Error())
+			return ctrl.Result{}, err
+		}
+	} else if client.IgnoreNotFound(err) != nil {
+		return ctrl.Result{}, err
 	}
-	if err != nil {
-		return ctrl.Result{}, fmt.Errorf("failed to get payment: %v", err)
+
+	payment := &userv1.Payment{}
+	if err := r.Get(ctx, client.ObjectKey{Namespace: req.Namespace, Name: req.Name}, payment); err != nil {
+		return ctrl.Result{}, client.IgnoreNotFound(err)
 	}
 	if payment.Spec.UserID == "" || payment.Spec.Amount == 0 {
 		return ctrl.Result{}, fmt.Errorf("payment is invalid: %v", payment)
@@ -90,31 +91,16 @@ func (r *AccountReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 		return ctrl.Result{}, nil
 	}

-	account := &userv1.Account{}
-	account.Name = payment.Spec.UserID
-	account.Namespace = helper.GetDefaultNamespace()
-	err = r.Get(ctx, client.ObjectKey{Namespace: helper.GetDefaultNamespace(), Name: account.Name}, account)
-	if errors.IsNotFound(err) {
-		account.Status.Balance = 0
-		account.Status.ChargeList = []userv1.Charge{}
-		logger.Info("create account", "account", account)
-		if err := r.Create(ctx, account); err != nil {
-			return ctrl.Result{}, fmt.Errorf("create account failed: %v", err)
-		}
-	} else if err != nil {
+	account, err := r.syncAccount(ctx, payment.Spec.UserID, r.AccountSystemNameSpace, payment.Namespace)
+	if err != nil {
 		return ctrl.Result{}, fmt.Errorf("get account failed: %v", err)
 	}

-	// add role get account permission
-	if err := r.syncRoleAndRoleBinding(ctx, payment.Spec.UserID, payment.Namespace); err != nil {
-		return ctrl.Result{}, fmt.Errorf("sync role and rolebinding failed: %v", err)
-	}
-
 	status, err := pay.QueryOrder(payment.Status.TradeNO)
 	if err != nil {
 		return ctrl.Result{}, fmt.Errorf("query order failed: %v", err)
 	}
-	logger.V(1).Info("query order status", "status", status)
+	r.Logger.V(1).Info("query order status", "status", status)
 	switch status {
 	case pay.StatusSuccess:
 		account.Status.ChargeList = append(account.Status.ChargeList, userv1.Charge{
@@ -146,11 +132,30 @@ func (r *AccountReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 	return ctrl.Result{}, nil
 }

+func (r *AccountReconciler) syncAccount(ctx context.Context, name, accountNamespace string, userNamespace string) (*userv1.Account, error) {
+	account := userv1.Account{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: accountNamespace,
+		},
+	}
+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &account, func() error {
+		return nil
+	}); err != nil {
+		return nil, err
+	}
+	// add role get account permission
+	if err := r.syncRoleAndRoleBinding(ctx, name, userNamespace); err != nil {
+		return nil, fmt.Errorf("sync role and rolebinding failed: %v", err)
+	}
+	return &account, nil
+}
+
 func (r *AccountReconciler) syncRoleAndRoleBinding(ctx context.Context, name, namespace string) error {
 	role := rbacV1.Role{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "userAccountRole-" + name,
-			Namespace: helper.GetDefaultNamespace(),
+			Namespace: r.AccountSystemNameSpace,
 		},
 	}
 	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &role, func() error {
@@ -169,7 +174,7 @@ func (r *AccountReconciler) syncRoleAndRoleBinding(ctx context.Context, name, na
 	roleBinding := rbacV1.RoleBinding{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "userAccountRoleBinding-" + name,
-			Namespace: helper.GetDefaultNamespace(),
+			Namespace: r.AccountSystemNameSpace,
 		},
 	}
 	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, &roleBinding, func() error {
@@ -205,13 +210,52 @@ func (r *AccountReconciler) DeletePayment(ctx context.Context) error {
 	return nil
 }

+func (r *AccountReconciler) updateDeductionBalance(ctx context.Context, accountBalance *userv1.AccountBalance) error {
+	if accountBalance.Status.Status == meteringv1.Complete {
+		return nil
+	}
+
+	r.Logger.V(1).Info("enter deduction balance", "accountBalanceName", accountBalance.Name, "accountBalanceNameSpace", accountBalance.Namespace, ".Spec", accountBalance.Spec, "status", accountBalance.Status)
+	account, err := r.syncAccount(ctx, accountBalance.Spec.Owner, r.AccountSystemNameSpace, "ns-"+accountBalance.Spec.Owner)
+	if err != nil {
+		r.Logger.Error(err, err.Error())
+		return err
+	}
+
+	account.Status.DeductionBalance += accountBalance.Spec.Amount
+	account.Status.ChargeList = append(account.Status.ChargeList, userv1.Charge{
+		Amount:             accountBalance.Spec.Amount,
+		Time:               metav1.Now(),
+		Status:             string(accountBalance.Status.Status),
+		AccountBalanceName: accountBalance.Name,
+	})
+
+	if err := r.Status().Update(ctx, account); err != nil {
+		r.Logger.Error(err, err.Error())
+		return err
+	}
+
+	accountBalance.Status.Status = meteringv1.Complete
+	if err := r.Status().Update(ctx, accountBalance); err != nil {
+		r.Logger.Error(err, err.Error())
+		return err
+	}
+	return nil
+}
+
 // SetupWithManager sets up the controller with the Manager.
 func (r *AccountReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	const controllerName = "account_controller"
 	r.Logger = ctrl.Log.WithName(controllerName)
 	r.Logger.V(1).Info("init reconcile controller account")
+
+	r.AccountSystemNameSpace = os.Getenv(ACCOUNTNAMESPACEENV)
+	if r.AccountSystemNameSpace == "" {
+		r.AccountSystemNameSpace = DEFAULTACCOUNTNAMESPACE
+	}
 	return ctrl.NewControllerManagedBy(mgr).
 		For(&userv1.Account{}).
 		Watches(&source.Kind{Type: &userv1.Payment{}}, &handler.EnqueueRequestForObject{}).
+		Watches(&source.Kind{Type: &userv1.AccountBalance{}}, &handler.EnqueueRequestForObject{}).
 		Complete(r)
 }
@@ -0,0 +1,63 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.10.0
+  creationTimestamp: null
+  name: accountbalances.user.sealos.io
+spec:
+  group: user.sealos.io
+  names:
+    kind: AccountBalance
+    listKind: AccountBalanceList
+    plural: accountbalances
+    singular: accountbalance
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.amount
+      name: Amount
+      type: integer
+    name: v1
+    schema:
+      openAPIV3Schema:
+        description: AccountBalance is the Schema for the accountbalances API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: AccountBalanceSpec defines the desired state of AccountBalance
+            properties:
+              amount:
+                format: int64
+                type: integer
+              owner:
+                type: string
+              timeStamp:
+                format: int64
+                type: integer
+            required:
+            - owner
+            type: object
+          status:
+            description: AccountBalanceStatus defines the observed state of AccountBalance
+            properties:
+              status:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
@@ -39,12 +39,10 @@ spec:
            description: AccountStatus defines the observed state of Account
            properties:
              balance:
+                description: Recharge amount
                format: int64
                type: integer
              chargeList:
-                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state
-                  of cluster Important: Run "make" to regenerate code after modifying
-                  this file'
                items:
                  properties:
                    balance:
@@ -59,6 +57,10 @@ spec:
                      type: string
                  type: object
                type: array
+              deductionBalance:
+                description: Deduction amount
+                format: int64
+                type: integer
            type: object
        type: object
    served: true
@@ -7,6 +7,7 @@ resources:
 - bases/user.sealos.io_usergroupbindings.yaml
 - bases/user.sealos.io_accounts.yaml
 - bases/user.sealos.io_payments.yaml
+- bases/user.sealos.io_accountbalances.yaml
 #+kubebuilder:scaffold:crdkustomizeresource

 patchesStrategicMerge:
@@ -20,6 +21,7 @@ patchesStrategicMerge:
 #- patches/webhook_in_usergroupbindings.yaml
 #- patches/webhook_in_accounts.yaml
 #- patches/webhook_in_payments.yaml
+#- patches/webhook_in_accountbalances.yaml
 #+kubebuilder:scaffold:crdkustomizewebhookpatch

 # [CERTMANAGER] To enable cert-manager, uncomment all the sections with [CERTMANAGER] prefix.
@@ -32,6 +34,7 @@ patchesStrategicMerge:
 #- patches/cainjection_in_usergroupbindings.yaml
 #- patches/cainjection_in_accounts.yaml
 #- patches/cainjection_in_payments.yaml
+#- patches/cainjection_in_accountbalances.yaml
 #+kubebuilder:scaffold:crdkustomizecainjectionpatch

 # the following config is for teaching kustomize how to do kustomization for CRDs.
@@ -0,0 +1,7 @@
+# The following patch adds a directive for certmanager to inject CA into the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: accountbalances.user.sealos.io
@@ -0,0 +1,16 @@
+# The following patch enables a conversion webhook for the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: accountbalances.user.sealos.io
+spec:
+  conversion:
+    strategy: Webhook
+    webhook:
+      clientConfig:
+        service:
+          namespace: system
+          name: webhook-service
+          path: /convert
+      conversionReviewVersions:
+      - v1
@@ -48,3 +48,5 @@ spec:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
+          - name: ACCOUNT_NAMESPACE
+            value: "sealos-system"
@@ -0,0 +1,24 @@
+# permissions for end users to edit accountbalances.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: accountbalance-editor-role
+rules:
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances/status
+  verbs:
+  - get
@@ -0,0 +1,20 @@
+# permissions for end users to view accountbalances.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: accountbalance-viewer-role
+rules:
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances/status
+  verbs:
+  - get
@@ -127,6 +127,18 @@ rules:
  - patch
  - update
  - watch
+- apiGroups:
+  - user.sealos.io
+  resources:
+  - accountbalances
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
 - apiGroups:
  - user.sealos.io
  resources:
@@ -0,0 +1,6 @@
+apiVersion: user.sealos.io/v1
+kind: AccountBalance
+metadata:
+  name: accountbalance-sample
+spec:
+  # TODO(user): Add fields here